Nov 01 05:42:38.342733 ip-10-0-143-195 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 01 05:42:38.342747 ip-10-0-143-195 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 01 05:42:38.342757 ip-10-0-143-195 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 01 05:42:38.343107 ip-10-0-143-195 systemd[1]: Failed to start Kubernetes Kubelet. Nov 01 05:42:48.591628 ip-10-0-143-195 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 01 05:42:48.591644 ip-10-0-143-195 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 2f0bdd67682e41f4a088ce025a98d64e -- Nov 01 05:44:36.409293 ip-10-0-143-195 systemd[1]: Starting Kubernetes Kubelet... Nov 01 05:44:36.830238 ip-10-0-143-195 kubenswrapper[2586]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 05:44:36.830238 ip-10-0-143-195 kubenswrapper[2586]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 05:44:36.830238 ip-10-0-143-195 kubenswrapper[2586]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 05:44:36.830238 ip-10-0-143-195 kubenswrapper[2586]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 05:44:36.835894 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.835807 2586 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 05:44:36.835894 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.835891 2586 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 05:44:36.835894 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.835895 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 05:44:36.835894 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.835899 2586 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 05:44:36.835894 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.835902 2586 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 05:44:36.836101 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.835905 2586 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 05:44:36.836101 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.835908 2586 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 05:44:36.836101 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.835911 2586 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 05:44:36.836101 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.835913 2586 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 05:44:36.836101 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.835916 2586 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 05:44:36.836101 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.835920 2586 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 05:44:36.836101 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.835923 2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 05:44:36.836101 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.835925 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 05:44:36.836101 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.835928 2586 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 05:44:36.836101 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.835933 2586 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 05:44:36.836101 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.835937 2586 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 05:44:36.836101 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.835940 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 05:44:36.836101 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.835948 2586 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 05:44:36.836101 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.835951 2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 05:44:36.836101 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.835954 2586 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 05:44:36.836101 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.835957 2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 05:44:36.836101 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.835959 2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 05:44:36.836101 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.835962 2586 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 05:44:36.836101 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.835964 2586 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 05:44:36.836541 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.835967 2586 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 05:44:36.836541 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.835970 2586 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 05:44:36.836541 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.835972 2586 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 05:44:36.836541 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.835975 2586 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 05:44:36.836541 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.835978 2586 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 05:44:36.836541 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.835980 2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 05:44:36.836541 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.835999 2586 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 05:44:36.836541 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836002 2586 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 05:44:36.836541 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836005 2586 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 05:44:36.836541 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836008 2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 05:44:36.836541 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836010 2586 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 05:44:36.836541 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836013 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 05:44:36.836541 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836016 2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 05:44:36.836541 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836019 2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 05:44:36.836541 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836022 2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 05:44:36.836541 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836025 2586 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 05:44:36.836541 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836028 2586 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 05:44:36.836541 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836032 2586 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 05:44:36.836541 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836034 2586 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 05:44:36.836541 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836037 2586 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 05:44:36.837026 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836040 2586 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 05:44:36.837026 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836042 2586 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 05:44:36.837026 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836045 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 05:44:36.837026 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836047 2586 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 05:44:36.837026 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836050 2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 05:44:36.837026 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836053 2586 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 05:44:36.837026 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836056 2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 05:44:36.837026 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836058 2586 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 05:44:36.837026 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836061 2586 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 05:44:36.837026 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836064 2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 05:44:36.837026 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836066 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 05:44:36.837026 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836069 2586 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 05:44:36.837026 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836071 2586 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 05:44:36.837026 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836074 2586 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 05:44:36.837026 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836076 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 05:44:36.837026 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836081 2586 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 05:44:36.837026 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836085 2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 05:44:36.837026 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836088 2586 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 05:44:36.837026 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836090 2586 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 05:44:36.837477 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836093 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 05:44:36.837477 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836096 2586 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 05:44:36.837477 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836098 2586 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 05:44:36.837477 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836101 2586 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 05:44:36.837477 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836104 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 05:44:36.837477 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836107 2586 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 05:44:36.837477 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836109 2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 05:44:36.837477 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836113 2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 05:44:36.837477 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836116 2586 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 05:44:36.837477 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836118 2586 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 05:44:36.837477 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836121 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 05:44:36.837477 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836125 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 05:44:36.837477 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836127 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 05:44:36.837477 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836130 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 05:44:36.837477 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836133 2586 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 05:44:36.837477 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836135 2586 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 05:44:36.837477 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836138 2586 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 05:44:36.837477 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836141 2586 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 05:44:36.837477 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836143 2586 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 05:44:36.837921 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836146 2586 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 05:44:36.837921 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836149 2586 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 05:44:36.837921 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836151 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 05:44:36.837921 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836154 2586 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 05:44:36.837921 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836156 2586 feature_gate.go:386] unrecognized feature gate: Example Nov 01 05:44:36.837921 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836159 2586 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 05:44:36.837921 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836161 2586 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 05:44:36.837921 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836166 2586 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 05:44:36.837921 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836168 2586 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 05:44:36.837921 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836171 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 05:44:36.837921 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836173 2586 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 05:44:36.837921 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836176 2586 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 05:44:36.837921 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836178 2586 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 05:44:36.837921 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836181 2586 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 05:44:36.837921 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836183 2586 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 05:44:36.837921 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836186 2586 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 05:44:36.837921 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836188 2586 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 05:44:36.837921 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836191 2586 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 05:44:36.837921 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836194 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 05:44:36.837921 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836196 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 05:44:36.838437 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836199 2586 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 05:44:36.838437 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836202 2586 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 05:44:36.838437 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836205 2586 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 05:44:36.838437 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.836207 2586 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 05:44:36.838437 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837060 2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 05:44:36.838437 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837070 2586 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 05:44:36.838437 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837073 2586 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 05:44:36.838437 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837076 2586 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 05:44:36.838437 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837079 2586 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 05:44:36.838437 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837082 2586 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 05:44:36.838437 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837086 2586 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 05:44:36.838437 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837088 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 05:44:36.838437 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837091 2586 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 05:44:36.838437 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837094 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 05:44:36.838437 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837096 2586 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 05:44:36.838437 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837099 2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 05:44:36.838437 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837102 2586 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 05:44:36.838437 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837104 2586 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 05:44:36.838437 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837107 2586 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 05:44:36.838437 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837109 2586 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 05:44:36.838911 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837112 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 05:44:36.838911 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837115 2586 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 05:44:36.838911 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837118 2586 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 05:44:36.838911 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837121 2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 05:44:36.838911 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837123 2586 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 05:44:36.838911 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837126 2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 05:44:36.838911 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837128 2586 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 05:44:36.838911 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837131 2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 05:44:36.838911 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837134 2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 05:44:36.838911 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837136 2586 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 05:44:36.838911 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837139 2586 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 05:44:36.838911 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837142 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 05:44:36.838911 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837145 2586 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 05:44:36.838911 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837148 2586 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 05:44:36.838911 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837150 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 05:44:36.838911 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837153 2586 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 05:44:36.838911 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837156 2586 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 05:44:36.838911 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837158 2586 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 05:44:36.838911 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837161 2586 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 05:44:36.838911 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837164 2586 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 05:44:36.839418 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837167 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 05:44:36.839418 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837170 2586 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 05:44:36.839418 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837172 2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 05:44:36.839418 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837175 2586 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 05:44:36.839418 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837179 2586 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 05:44:36.839418 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837182 2586 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 05:44:36.839418 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837186 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 05:44:36.839418 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837188 2586 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 05:44:36.839418 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837191 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 05:44:36.839418 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837193 2586 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 05:44:36.839418 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837196 2586 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 05:44:36.839418 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837198 2586 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 05:44:36.839418 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837201 2586 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 05:44:36.839418 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837204 2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 05:44:36.839418 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837206 2586 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 05:44:36.839418 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837209 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 05:44:36.839418 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837211 2586 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 05:44:36.839418 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837214 2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 05:44:36.839418 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837217 2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 05:44:36.839869 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837220 2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 05:44:36.839869 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837222 2586 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 05:44:36.839869 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837224 2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 05:44:36.839869 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837228 2586 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 05:44:36.839869 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837231 2586 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 05:44:36.839869 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837233 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 05:44:36.839869 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837236 2586 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 05:44:36.839869 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837239 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 05:44:36.839869 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837241 2586 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 05:44:36.839869 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837244 2586 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 05:44:36.839869 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837246 2586 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 05:44:36.839869 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837249 2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 05:44:36.839869 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837252 2586 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 05:44:36.839869 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837255 2586 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 05:44:36.839869 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837258 2586 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 05:44:36.839869 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837261 2586 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 05:44:36.839869 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837263 2586 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 05:44:36.839869 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837266 2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 05:44:36.839869 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837268 2586 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 05:44:36.839869 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837271 2586 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 05:44:36.840355 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837274 2586 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 05:44:36.840355 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837276 2586 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 05:44:36.840355 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837279 2586 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 05:44:36.840355 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837281 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 05:44:36.840355 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837284 2586 feature_gate.go:386] unrecognized feature gate: Example Nov 01 05:44:36.840355 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837286 2586 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 05:44:36.840355 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837289 2586 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 05:44:36.840355 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837292 2586 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 05:44:36.840355 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837294 2586 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 05:44:36.840355 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837297 2586 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 05:44:36.840355 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837299 2586 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 05:44:36.840355 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837302 2586 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 05:44:36.840355 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837305 2586 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 05:44:36.840355 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837311 2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 05:44:36.840355 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837314 2586 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 05:44:36.840355 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837317 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 05:44:36.840355 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837319 2586 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 05:44:36.840355 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837322 2586 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 05:44:36.840355 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837324 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 05:44:36.840355 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837329 2586 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 05:44:36.840820 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837333 2586 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 05:44:36.840820 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837335 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 05:44:36.840820 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837338 2586 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 05:44:36.840820 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837341 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 05:44:36.840820 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837343 2586 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 05:44:36.840820 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837352 2586 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 05:44:36.840820 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837355 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 05:44:36.840820 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837357 2586 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 05:44:36.840820 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837360 2586 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 05:44:36.840820 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837362 2586 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 05:44:36.840820 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837365 2586 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 05:44:36.840820 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837953 2586 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 05:44:36.840820 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837957 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 05:44:36.840820 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837960 2586 feature_gate.go:386] unrecognized feature gate: Example Nov 01 05:44:36.840820 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837963 2586 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 05:44:36.840820 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837966 2586 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 05:44:36.840820 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837968 2586 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 05:44:36.840820 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837971 2586 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 05:44:36.840820 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837974 2586 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 05:44:36.840820 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837976 2586 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 05:44:36.841300 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837979 2586 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 05:44:36.841300 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.837981 2586 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 05:44:36.841300 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838001 2586 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 05:44:36.841300 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838004 2586 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 05:44:36.841300 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838007 2586 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 05:44:36.841300 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838010 2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 05:44:36.841300 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838013 2586 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 05:44:36.841300 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838016 2586 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 05:44:36.841300 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838018 2586 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 05:44:36.841300 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838021 2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 05:44:36.841300 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838023 2586 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 05:44:36.841300 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838026 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 05:44:36.841300 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838029 2586 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 05:44:36.841300 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838031 2586 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 05:44:36.841300 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838034 2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 05:44:36.841300 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838036 2586 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 05:44:36.841300 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838039 2586 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 05:44:36.841300 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838049 2586 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 05:44:36.841300 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838052 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 05:44:36.841300 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838055 2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 05:44:36.841891 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838057 2586 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 05:44:36.841891 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838059 2586 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 05:44:36.841891 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838062 2586 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 05:44:36.841891 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838065 2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 05:44:36.841891 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838067 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 05:44:36.841891 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838070 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 05:44:36.841891 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838074 2586 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 05:44:36.841891 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838077 2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 05:44:36.841891 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838080 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 05:44:36.841891 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838083 2586 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 05:44:36.841891 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838085 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 05:44:36.841891 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838088 2586 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 05:44:36.841891 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838090 2586 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 05:44:36.841891 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838093 2586 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 05:44:36.841891 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838095 2586 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 05:44:36.841891 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838098 2586 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 05:44:36.841891 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838100 2586 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 05:44:36.841891 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838103 2586 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 05:44:36.842576 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838108 2586 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 05:44:36.842576 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838111 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 05:44:36.842576 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838115 2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 05:44:36.842576 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838118 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 05:44:36.842576 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838123 2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 05:44:36.842576 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838126 2586 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 05:44:36.842576 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838128 2586 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 05:44:36.842576 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838131 2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 05:44:36.842576 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838133 2586 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 05:44:36.842576 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838136 2586 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 05:44:36.842576 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838139 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 05:44:36.842576 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838141 2586 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 05:44:36.842576 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838149 2586 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 05:44:36.842576 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838152 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 05:44:36.842576 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838155 2586 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 05:44:36.842576 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838158 2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 05:44:36.842576 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838160 2586 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 05:44:36.842576 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838163 2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 05:44:36.842576 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838165 2586 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 05:44:36.843237 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838168 2586 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 05:44:36.843237 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838170 2586 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 05:44:36.843237 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838173 2586 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 05:44:36.843237 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838175 2586 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 05:44:36.843237 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838178 2586 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 05:44:36.843237 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838181 2586 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 05:44:36.843237 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838183 2586 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 05:44:36.843237 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838186 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 05:44:36.843237 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838188 2586 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 05:44:36.843237 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838191 2586 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 05:44:36.843237 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838193 2586 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 05:44:36.843237 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838196 2586 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 05:44:36.843237 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838198 2586 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 05:44:36.843237 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838201 2586 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 05:44:36.843237 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838204 2586 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 05:44:36.843237 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838206 2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 05:44:36.843237 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838209 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 05:44:36.843237 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838211 2586 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 05:44:36.843237 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838214 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 05:44:36.843237 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838216 2586 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 05:44:36.843791 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838219 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 05:44:36.843791 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838221 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 05:44:36.843791 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838224 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 05:44:36.843791 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838226 2586 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 05:44:36.843791 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838229 2586 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 05:44:36.843791 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838231 2586 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 05:44:36.843791 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838239 2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 05:44:36.843791 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838242 2586 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 05:44:36.843791 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838245 2586 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 05:44:36.843791 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838247 2586 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 05:44:36.843791 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838250 2586 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 05:44:36.843791 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838252 2586 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 05:44:36.843791 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838255 2586 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 05:44:36.843791 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838258 2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 05:44:36.843791 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838260 2586 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 05:44:36.843791 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838263 2586 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 05:44:36.843791 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838265 2586 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 05:44:36.843791 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838267 2586 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 05:44:36.843791 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838270 2586 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 05:44:36.843791 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.838272 2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 05:44:36.844377 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839267 2586 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 05:44:36.844377 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839277 2586 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 05:44:36.844377 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839287 2586 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 05:44:36.844377 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839292 2586 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 05:44:36.844377 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839297 2586 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 05:44:36.844377 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839301 2586 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 05:44:36.844377 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839306 2586 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 05:44:36.844377 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839310 2586 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 05:44:36.844377 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839313 2586 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 05:44:36.844377 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839316 2586 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 05:44:36.844377 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839320 2586 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 05:44:36.844377 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839324 2586 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 05:44:36.844377 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839327 2586 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 05:44:36.844377 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839329 2586 flags.go:64] FLAG: --cgroup-root="" Nov 01 05:44:36.844377 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839333 2586 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 05:44:36.844377 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839336 2586 flags.go:64] FLAG: --client-ca-file="" Nov 01 05:44:36.844377 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839339 2586 flags.go:64] FLAG: --cloud-provider="external" Nov 01 05:44:36.844377 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839341 2586 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 05:44:36.844377 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839349 2586 flags.go:64] FLAG: --cluster-domain="" Nov 01 05:44:36.844377 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839357 2586 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 05:44:36.844377 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839361 2586 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 05:44:36.844377 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839364 2586 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 05:44:36.844377 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839367 2586 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 05:44:36.844377 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839372 2586 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 05:44:36.844929 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839375 2586 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 05:44:36.844929 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839378 2586 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 05:44:36.844929 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839381 2586 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 05:44:36.844929 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839384 2586 flags.go:64] FLAG: --contention-profiling="false" Nov 01 05:44:36.844929 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839387 2586 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 05:44:36.844929 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839390 2586 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 05:44:36.844929 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839393 2586 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 05:44:36.844929 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839396 2586 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 05:44:36.844929 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839401 2586 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 05:44:36.844929 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839404 2586 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 05:44:36.844929 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839406 2586 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 05:44:36.844929 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839409 2586 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 05:44:36.844929 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839415 2586 flags.go:64] FLAG: --enable-server="true" Nov 01 05:44:36.844929 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839418 2586 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 05:44:36.844929 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839425 2586 flags.go:64] FLAG: --event-burst="100" Nov 01 05:44:36.844929 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839429 2586 flags.go:64] FLAG: --event-qps="50" Nov 01 05:44:36.844929 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839432 2586 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 05:44:36.844929 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839435 2586 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 05:44:36.844929 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839438 2586 flags.go:64] FLAG: --eviction-hard="" Nov 01 05:44:36.844929 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839442 2586 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 05:44:36.844929 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839445 2586 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 05:44:36.844929 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839448 2586 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 05:44:36.844929 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839452 2586 flags.go:64] FLAG: --eviction-soft="" Nov 01 05:44:36.844929 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839455 2586 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 05:44:36.844929 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839458 2586 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 05:44:36.845574 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839461 2586 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 05:44:36.845574 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839464 2586 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 05:44:36.845574 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839468 2586 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 05:44:36.845574 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839476 2586 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 05:44:36.845574 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839479 2586 flags.go:64] FLAG: --feature-gates="" Nov 01 05:44:36.845574 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839484 2586 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 05:44:36.845574 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839487 2586 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 05:44:36.845574 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839490 2586 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 05:44:36.845574 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839494 2586 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 05:44:36.845574 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839497 2586 flags.go:64] FLAG: --healthz-port="10248" Nov 01 05:44:36.845574 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839500 2586 flags.go:64] FLAG: --help="false" Nov 01 05:44:36.845574 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839503 2586 flags.go:64] FLAG: --hostname-override="ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:44:36.845574 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839506 2586 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 05:44:36.845574 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839509 2586 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 05:44:36.845574 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839512 2586 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 05:44:36.845574 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839516 2586 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 05:44:36.845574 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839519 2586 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 05:44:36.845574 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839522 2586 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 05:44:36.845574 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839525 2586 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 05:44:36.845574 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839529 2586 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 05:44:36.845574 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839532 2586 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 05:44:36.845574 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839535 2586 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 05:44:36.845574 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839539 2586 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 05:44:36.846146 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839542 2586 flags.go:64] FLAG: --kube-reserved="" Nov 01 05:44:36.846146 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839545 2586 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 05:44:36.846146 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839548 2586 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 05:44:36.846146 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839551 2586 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 05:44:36.846146 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839553 2586 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 05:44:36.846146 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839557 2586 flags.go:64] FLAG: --lock-file="" Nov 01 05:44:36.846146 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839560 2586 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 05:44:36.846146 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839562 2586 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 05:44:36.846146 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839571 2586 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 05:44:36.846146 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839577 2586 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 05:44:36.846146 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839581 2586 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 05:44:36.846146 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839584 2586 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 05:44:36.846146 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839587 2586 flags.go:64] FLAG: --logging-format="text" Nov 01 05:44:36.846146 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839595 2586 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 05:44:36.846146 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839599 2586 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 05:44:36.846146 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839602 2586 flags.go:64] FLAG: --manifest-url="" Nov 01 05:44:36.846146 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839605 2586 flags.go:64] FLAG: --manifest-url-header="" Nov 01 05:44:36.846146 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839609 2586 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 05:44:36.846146 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839612 2586 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 05:44:36.846146 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839616 2586 flags.go:64] FLAG: --max-pods="110" Nov 01 05:44:36.846146 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839619 2586 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 05:44:36.846146 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839622 2586 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 05:44:36.846146 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839625 2586 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 05:44:36.846146 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839628 2586 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 05:44:36.846146 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839631 2586 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 05:44:36.846743 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839634 2586 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 05:44:36.846743 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839638 2586 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 05:44:36.846743 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839645 2586 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 05:44:36.846743 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839650 2586 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 05:44:36.846743 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839653 2586 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 05:44:36.846743 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839656 2586 flags.go:64] FLAG: --pod-cidr="" Nov 01 05:44:36.846743 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839659 2586 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 05:44:36.846743 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839662 2586 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 05:44:36.846743 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839665 2586 flags.go:64] FLAG: --pods-per-core="0" Nov 01 05:44:36.846743 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839668 2586 flags.go:64] FLAG: --port="10250" Nov 01 05:44:36.846743 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839671 2586 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 05:44:36.846743 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839674 2586 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-00708e3e63e636aaf" Nov 01 05:44:36.846743 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839677 2586 flags.go:64] FLAG: --qos-reserved="" Nov 01 05:44:36.846743 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839680 2586 flags.go:64] FLAG: --read-only-port="10255" Nov 01 05:44:36.846743 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839683 2586 flags.go:64] FLAG: --register-node="true" Nov 01 05:44:36.846743 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839686 2586 flags.go:64] FLAG: --register-with-taints="" Nov 01 05:44:36.846743 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839694 2586 flags.go:64] FLAG: --registry-burst="10" Nov 01 05:44:36.846743 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839697 2586 flags.go:64] FLAG: --registry-qps="5" Nov 01 05:44:36.846743 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839700 2586 flags.go:64] FLAG: --reserved-cpus="" Nov 01 05:44:36.846743 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839703 2586 flags.go:64] FLAG: --reserved-memory="" Nov 01 05:44:36.846743 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839707 2586 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 05:44:36.846743 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839716 2586 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 05:44:36.846743 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839720 2586 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 05:44:36.846743 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839723 2586 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 05:44:36.846743 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839726 2586 flags.go:64] FLAG: --runonce="false" Nov 01 05:44:36.846743 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839729 2586 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 05:44:36.847465 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839732 2586 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 05:44:36.847465 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839735 2586 flags.go:64] FLAG: --seccomp-default="false" Nov 01 05:44:36.847465 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839738 2586 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 05:44:36.847465 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839741 2586 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 05:44:36.847465 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839744 2586 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 05:44:36.847465 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839747 2586 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 05:44:36.847465 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839750 2586 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 05:44:36.847465 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839753 2586 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 05:44:36.847465 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839758 2586 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 05:44:36.847465 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839762 2586 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 05:44:36.847465 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839765 2586 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 05:44:36.847465 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839769 2586 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 05:44:36.847465 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839772 2586 flags.go:64] FLAG: --system-cgroups="" Nov 01 05:44:36.847465 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839775 2586 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 05:44:36.847465 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839780 2586 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 05:44:36.847465 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839783 2586 flags.go:64] FLAG: --tls-cert-file="" Nov 01 05:44:36.847465 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839786 2586 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 05:44:36.847465 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839793 2586 flags.go:64] FLAG: --tls-min-version="" Nov 01 05:44:36.847465 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839796 2586 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 05:44:36.847465 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839799 2586 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 05:44:36.847465 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839802 2586 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 05:44:36.847465 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839805 2586 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 05:44:36.847465 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839808 2586 flags.go:64] FLAG: --v="2" Nov 01 05:44:36.847465 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839812 2586 flags.go:64] FLAG: --version="false" Nov 01 05:44:36.847465 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839816 2586 flags.go:64] FLAG: --vmodule="" Nov 01 05:44:36.848142 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839820 2586 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 05:44:36.848142 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.839823 2586 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 05:44:36.848142 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.839995 2586 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 05:44:36.848142 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840012 2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 05:44:36.848142 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840016 2586 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 05:44:36.848142 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840019 2586 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 05:44:36.848142 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840022 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 05:44:36.848142 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840025 2586 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 05:44:36.848142 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840027 2586 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 05:44:36.848142 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840030 2586 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 05:44:36.848142 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840032 2586 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 05:44:36.848142 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840035 2586 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 05:44:36.848142 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840038 2586 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 05:44:36.848142 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840040 2586 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 05:44:36.848142 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840043 2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 05:44:36.848142 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840047 2586 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 05:44:36.848142 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840051 2586 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 05:44:36.848142 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840054 2586 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 05:44:36.848142 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840056 2586 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 05:44:36.848142 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840059 2586 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 05:44:36.848623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840062 2586 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 05:44:36.848623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840064 2586 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 05:44:36.848623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840066 2586 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 05:44:36.848623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840069 2586 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 05:44:36.848623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840071 2586 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 05:44:36.848623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840074 2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 05:44:36.848623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840076 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 05:44:36.848623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840079 2586 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 05:44:36.848623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840081 2586 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 05:44:36.848623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840084 2586 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 05:44:36.848623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840086 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 05:44:36.848623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840089 2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 05:44:36.848623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840091 2586 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 05:44:36.848623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840094 2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 05:44:36.848623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840097 2586 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 05:44:36.848623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840099 2586 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 05:44:36.848623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840108 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 05:44:36.848623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840110 2586 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 05:44:36.848623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840113 2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 05:44:36.848623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840116 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 05:44:36.849140 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840118 2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 05:44:36.849140 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840121 2586 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 05:44:36.849140 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840123 2586 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 05:44:36.849140 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840126 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 05:44:36.849140 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840128 2586 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 05:44:36.849140 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840131 2586 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 05:44:36.849140 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840133 2586 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 05:44:36.849140 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840137 2586 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 05:44:36.849140 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840141 2586 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 05:44:36.849140 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840144 2586 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 05:44:36.849140 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840147 2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 05:44:36.849140 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840150 2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 05:44:36.849140 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840152 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 05:44:36.849140 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840154 2586 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 05:44:36.849140 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840157 2586 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 05:44:36.849140 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840159 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 05:44:36.849140 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840162 2586 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 05:44:36.849140 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840164 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 05:44:36.849140 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840167 2586 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 05:44:36.850170 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840170 2586 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 05:44:36.850170 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840172 2586 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 05:44:36.850170 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840175 2586 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 05:44:36.850170 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840177 2586 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 05:44:36.850170 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840180 2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 05:44:36.850170 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840182 2586 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 05:44:36.850170 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840185 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 05:44:36.850170 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840187 2586 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 05:44:36.850170 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840190 2586 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 05:44:36.850170 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840193 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 05:44:36.850170 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840201 2586 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 05:44:36.850170 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840203 2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 05:44:36.850170 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840206 2586 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 05:44:36.850170 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840208 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 05:44:36.850170 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840211 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 05:44:36.850170 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840213 2586 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 05:44:36.850170 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840216 2586 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 05:44:36.850170 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840219 2586 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 05:44:36.850170 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840221 2586 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 05:44:36.850623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840224 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 05:44:36.850623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840228 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 05:44:36.850623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840233 2586 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 05:44:36.850623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840236 2586 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 05:44:36.850623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840239 2586 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 05:44:36.850623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840242 2586 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 05:44:36.850623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840245 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 05:44:36.850623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840247 2586 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 05:44:36.850623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840250 2586 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 05:44:36.850623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840253 2586 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 05:44:36.850623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840255 2586 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 05:44:36.850623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840258 2586 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 05:44:36.850623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840260 2586 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 05:44:36.850623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840263 2586 feature_gate.go:386] unrecognized feature gate: Example Nov 01 05:44:36.850623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840266 2586 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 05:44:36.850623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840268 2586 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 05:44:36.850623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840271 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 05:44:36.850623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840273 2586 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 05:44:36.850623 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840276 2586 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 05:44:36.851184 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840278 2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 05:44:36.851184 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840281 2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 05:44:36.851184 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840283 2586 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 05:44:36.851184 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840287 2586 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 05:44:36.851184 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840291 2586 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 05:44:36.851184 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840300 2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 05:44:36.851184 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840304 2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 05:44:36.851184 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840307 2586 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 05:44:36.851184 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840310 2586 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 05:44:36.851184 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840312 2586 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 05:44:36.851184 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840315 2586 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 05:44:36.851184 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.840343 2586 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 05:44:36.851184 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840386 2586 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 05:44:36.851184 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840391 2586 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 05:44:36.851184 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840396 2586 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 05:44:36.851538 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840399 2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 05:44:36.851538 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840403 2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 05:44:36.851538 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840406 2586 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 05:44:36.851538 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840408 2586 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 05:44:36.851538 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840411 2586 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 05:44:36.851538 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840414 2586 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 05:44:36.851538 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840417 2586 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 05:44:36.851538 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840419 2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 05:44:36.851538 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840422 2586 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 05:44:36.851538 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840424 2586 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 05:44:36.851538 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840427 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 05:44:36.851538 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840430 2586 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 05:44:36.851538 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840432 2586 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 05:44:36.851538 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840435 2586 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 05:44:36.851538 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840437 2586 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 05:44:36.851538 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840440 2586 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 05:44:36.851538 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840442 2586 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 05:44:36.851538 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840445 2586 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 05:44:36.851538 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840447 2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 05:44:36.851538 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840450 2586 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 05:44:36.852023 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840452 2586 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 05:44:36.852023 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840455 2586 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 05:44:36.852023 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840457 2586 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 05:44:36.852023 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840465 2586 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 05:44:36.852023 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840468 2586 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 05:44:36.852023 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840470 2586 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 05:44:36.852023 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840473 2586 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 05:44:36.852023 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840475 2586 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 05:44:36.852023 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840478 2586 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 05:44:36.852023 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840480 2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 05:44:36.852023 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840483 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 05:44:36.852023 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840487 2586 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 05:44:36.852023 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840489 2586 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 05:44:36.852023 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840493 2586 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 05:44:36.852023 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840495 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 05:44:36.852023 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840498 2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 05:44:36.852023 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840500 2586 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 05:44:36.852023 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840503 2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 05:44:36.852023 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840505 2586 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 05:44:36.852023 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840508 2586 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 05:44:36.852487 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840510 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 05:44:36.852487 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840513 2586 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 05:44:36.852487 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840515 2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 05:44:36.852487 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840518 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 05:44:36.852487 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840520 2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 05:44:36.852487 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840523 2586 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 05:44:36.852487 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840525 2586 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 05:44:36.852487 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840528 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 05:44:36.852487 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840530 2586 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 05:44:36.852487 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840533 2586 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 05:44:36.852487 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840535 2586 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 05:44:36.852487 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840538 2586 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 05:44:36.852487 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840540 2586 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 05:44:36.852487 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840543 2586 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 05:44:36.852487 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840546 2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 05:44:36.852487 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840548 2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 05:44:36.852487 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840558 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 05:44:36.852487 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840561 2586 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 05:44:36.852487 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840563 2586 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 05:44:36.852925 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840566 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 05:44:36.852925 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840568 2586 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 05:44:36.852925 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840571 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 05:44:36.852925 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840573 2586 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 05:44:36.852925 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840579 2586 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 05:44:36.852925 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840581 2586 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 05:44:36.852925 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840585 2586 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 05:44:36.852925 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840588 2586 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 05:44:36.852925 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840590 2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 05:44:36.852925 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840593 2586 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 05:44:36.852925 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840595 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 05:44:36.852925 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840598 2586 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 05:44:36.852925 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840600 2586 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 05:44:36.852925 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840603 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 05:44:36.852925 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840605 2586 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 05:44:36.852925 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840608 2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 05:44:36.852925 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840610 2586 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 05:44:36.852925 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840613 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 05:44:36.852925 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840615 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 05:44:36.853419 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840618 2586 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 05:44:36.853419 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840620 2586 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 05:44:36.853419 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840623 2586 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 05:44:36.853419 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840626 2586 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 05:44:36.853419 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840634 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 05:44:36.853419 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840637 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 05:44:36.853419 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840640 2586 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 05:44:36.853419 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840644 2586 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 05:44:36.853419 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840647 2586 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 05:44:36.853419 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840650 2586 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 05:44:36.853419 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840653 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 05:44:36.853419 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840661 2586 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 05:44:36.853419 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840664 2586 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 05:44:36.853419 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840666 2586 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 05:44:36.853419 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840669 2586 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 05:44:36.853419 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840671 2586 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 05:44:36.853419 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840674 2586 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 05:44:36.853419 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840679 2586 feature_gate.go:386] unrecognized feature gate: Example Nov 01 05:44:36.853419 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840681 2586 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 05:44:36.853856 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840685 2586 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 05:44:36.853856 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840688 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 05:44:36.853856 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840690 2586 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 05:44:36.853856 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840693 2586 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 05:44:36.853856 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840695 2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 05:44:36.853856 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.840698 2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 05:44:36.853856 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.850438 2586 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 05:44:36.853856 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.850456 2586 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 05:44:36.853856 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.850473 2586 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 05:44:36.853856 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.850478 2586 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 05:44:36.853856 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850535 2586 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 05:44:36.853856 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850542 2586 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 05:44:36.853856 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850546 2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 05:44:36.853856 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850550 2586 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 05:44:36.853856 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850553 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 05:44:36.853856 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850556 2586 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 05:44:36.853856 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850560 2586 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 05:44:36.853856 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850562 2586 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 05:44:36.853856 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850565 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 05:44:36.854332 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850568 2586 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 05:44:36.854332 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850570 2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 05:44:36.854332 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850573 2586 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 05:44:36.854332 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850576 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 05:44:36.854332 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850578 2586 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 05:44:36.854332 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850581 2586 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 05:44:36.854332 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850584 2586 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 05:44:36.854332 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850586 2586 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 05:44:36.854332 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850604 2586 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 05:44:36.854332 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850608 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 05:44:36.854332 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850612 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 05:44:36.854332 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850615 2586 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 05:44:36.854332 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850618 2586 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 05:44:36.854332 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850620 2586 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 05:44:36.854332 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850623 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 05:44:36.854332 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850626 2586 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 05:44:36.854332 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850629 2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 05:44:36.854332 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850632 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 05:44:36.854332 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850639 2586 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 05:44:36.854776 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850642 2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 05:44:36.854776 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850646 2586 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 05:44:36.854776 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850649 2586 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 05:44:36.854776 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850651 2586 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 05:44:36.854776 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850654 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 05:44:36.854776 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850657 2586 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 05:44:36.854776 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850660 2586 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 05:44:36.854776 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850662 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 05:44:36.854776 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850665 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 05:44:36.854776 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850667 2586 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 05:44:36.854776 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850670 2586 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 05:44:36.854776 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850673 2586 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 05:44:36.854776 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850675 2586 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 05:44:36.854776 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850678 2586 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 05:44:36.854776 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850680 2586 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 05:44:36.854776 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850683 2586 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 05:44:36.854776 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850685 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 05:44:36.854776 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850688 2586 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 05:44:36.854776 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850691 2586 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 05:44:36.854776 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850694 2586 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 05:44:36.855263 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850696 2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 05:44:36.855263 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850699 2586 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 05:44:36.855263 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850701 2586 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 05:44:36.855263 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850704 2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 05:44:36.855263 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850706 2586 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 05:44:36.855263 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850709 2586 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 05:44:36.855263 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850711 2586 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 05:44:36.855263 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850715 2586 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 05:44:36.855263 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850719 2586 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 05:44:36.855263 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850723 2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 05:44:36.855263 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850726 2586 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 05:44:36.855263 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850729 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 05:44:36.855263 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850731 2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 05:44:36.855263 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850735 2586 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 05:44:36.855263 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850738 2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 05:44:36.855263 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850740 2586 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 05:44:36.855263 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850743 2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 05:44:36.855263 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850746 2586 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 05:44:36.855263 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850749 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 05:44:36.855719 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850751 2586 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 05:44:36.855719 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850754 2586 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 05:44:36.855719 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850756 2586 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 05:44:36.855719 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850759 2586 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 05:44:36.855719 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850761 2586 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 05:44:36.855719 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850764 2586 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 05:44:36.855719 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850766 2586 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 05:44:36.855719 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850769 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 05:44:36.855719 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850772 2586 feature_gate.go:386] unrecognized feature gate: Example Nov 01 05:44:36.855719 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850775 2586 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 05:44:36.855719 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850778 2586 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 05:44:36.855719 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850789 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 05:44:36.855719 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850793 2586 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 05:44:36.855719 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850796 2586 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 05:44:36.855719 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850799 2586 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 05:44:36.855719 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850802 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 05:44:36.855719 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850804 2586 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 05:44:36.855719 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850807 2586 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 05:44:36.855719 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850809 2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 05:44:36.855719 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850812 2586 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 05:44:36.856303 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850815 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 05:44:36.856303 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850818 2586 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 05:44:36.856303 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850821 2586 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 05:44:36.856303 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850823 2586 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 05:44:36.856303 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850826 2586 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 05:44:36.856303 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850828 2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 05:44:36.856303 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850832 2586 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 05:44:36.856303 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850834 2586 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 05:44:36.856303 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850837 2586 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 05:44:36.856303 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850839 2586 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 05:44:36.856303 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850842 2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 05:44:36.856303 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850845 2586 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 05:44:36.856303 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850847 2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 05:44:36.856303 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850850 2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 05:44:36.856303 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850853 2586 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 05:44:36.856303 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850855 2586 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 05:44:36.856303 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850858 2586 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 05:44:36.856303 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850861 2586 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 05:44:36.856303 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.850863 2586 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 05:44:36.856754 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.850890 2586 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 05:44:36.856754 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851087 2586 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 05:44:36.856754 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851092 2586 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 05:44:36.856754 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851096 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 05:44:36.856754 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851099 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 05:44:36.856754 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851102 2586 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 05:44:36.856754 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851107 2586 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 05:44:36.856754 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851111 2586 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 05:44:36.856754 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851115 2586 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 05:44:36.856754 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851117 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 05:44:36.856754 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851120 2586 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 05:44:36.856754 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851123 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 05:44:36.856754 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851126 2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 05:44:36.856754 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851129 2586 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 05:44:36.856754 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851131 2586 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 05:44:36.857128 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851134 2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 05:44:36.857128 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851136 2586 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 05:44:36.857128 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851139 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 05:44:36.857128 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851141 2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 05:44:36.857128 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851144 2586 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 05:44:36.857128 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851146 2586 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 05:44:36.857128 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851149 2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 05:44:36.857128 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851152 2586 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 05:44:36.857128 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851154 2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 05:44:36.857128 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851157 2586 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 05:44:36.857128 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851160 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 05:44:36.857128 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851162 2586 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 05:44:36.857128 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851165 2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 05:44:36.857128 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851167 2586 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 05:44:36.857128 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851170 2586 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 05:44:36.857128 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851173 2586 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 05:44:36.857128 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851175 2586 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 05:44:36.857128 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851178 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 05:44:36.857128 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851180 2586 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 05:44:36.857128 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851183 2586 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 05:44:36.857600 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851185 2586 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 05:44:36.857600 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851188 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 05:44:36.857600 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851190 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 05:44:36.857600 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851193 2586 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 05:44:36.857600 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851196 2586 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 05:44:36.857600 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851199 2586 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 05:44:36.857600 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851202 2586 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 05:44:36.857600 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851204 2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 05:44:36.857600 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851207 2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 05:44:36.857600 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851210 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 05:44:36.857600 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851213 2586 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 05:44:36.857600 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851215 2586 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 05:44:36.857600 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851218 2586 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 05:44:36.857600 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851220 2586 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 05:44:36.857600 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851223 2586 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 05:44:36.857600 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851225 2586 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 05:44:36.857600 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851228 2586 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 05:44:36.857600 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851231 2586 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 05:44:36.857600 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851233 2586 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 05:44:36.858063 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851236 2586 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 05:44:36.858063 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851238 2586 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 05:44:36.858063 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851241 2586 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 05:44:36.858063 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851243 2586 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 05:44:36.858063 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851246 2586 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 05:44:36.858063 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851248 2586 feature_gate.go:386] unrecognized feature gate: Example Nov 01 05:44:36.858063 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851251 2586 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 05:44:36.858063 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851254 2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 05:44:36.858063 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851256 2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 05:44:36.858063 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851259 2586 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 05:44:36.858063 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851261 2586 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 05:44:36.858063 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851264 2586 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 05:44:36.858063 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851266 2586 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 05:44:36.858063 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851269 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 05:44:36.858063 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851271 2586 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 05:44:36.858063 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851273 2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 05:44:36.858063 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851276 2586 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 05:44:36.858063 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851278 2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 05:44:36.858063 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851281 2586 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 05:44:36.858063 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851284 2586 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 05:44:36.858544 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851287 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 05:44:36.858544 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851289 2586 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 05:44:36.858544 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851292 2586 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 05:44:36.858544 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851295 2586 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 05:44:36.858544 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851299 2586 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 05:44:36.858544 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851302 2586 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 05:44:36.858544 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851305 2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 05:44:36.858544 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851308 2586 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 05:44:36.858544 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851311 2586 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 05:44:36.858544 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851313 2586 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 05:44:36.858544 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851316 2586 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 05:44:36.858544 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851319 2586 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 05:44:36.858544 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851322 2586 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 05:44:36.858544 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851325 2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 05:44:36.858544 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851327 2586 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 05:44:36.858544 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851330 2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 05:44:36.858544 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851332 2586 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 05:44:36.858544 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851335 2586 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 05:44:36.858544 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851337 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 05:44:36.859057 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851340 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 05:44:36.859057 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851342 2586 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 05:44:36.859057 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851344 2586 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 05:44:36.859057 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851347 2586 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 05:44:36.859057 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851349 2586 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 05:44:36.859057 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851352 2586 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 05:44:36.859057 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851354 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 05:44:36.859057 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851357 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 05:44:36.859057 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851359 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 05:44:36.859057 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851362 2586 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 05:44:36.859057 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851364 2586 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 05:44:36.859057 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851367 2586 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 05:44:36.859057 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851370 2586 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 05:44:36.859057 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851373 2586 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 05:44:36.859057 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.851397 2586 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 05:44:36.859057 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851440 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 05:44:36.859428 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851444 2586 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 05:44:36.859428 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851447 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 05:44:36.859428 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851450 2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 05:44:36.859428 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851452 2586 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 05:44:36.859428 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851455 2586 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 05:44:36.859428 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851458 2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 05:44:36.859428 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851460 2586 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 05:44:36.859428 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851463 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 05:44:36.859428 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851465 2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 05:44:36.859428 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851468 2586 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 05:44:36.859428 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851471 2586 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 05:44:36.859428 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851473 2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 05:44:36.859428 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851476 2586 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 05:44:36.859428 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851478 2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 05:44:36.859428 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851481 2586 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 05:44:36.859428 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851483 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 05:44:36.859428 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851486 2586 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 05:44:36.859428 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851488 2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 05:44:36.859428 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851491 2586 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 05:44:36.859428 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851493 2586 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 05:44:36.859913 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851496 2586 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 05:44:36.859913 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851498 2586 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 05:44:36.859913 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851501 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 05:44:36.859913 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851503 2586 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 05:44:36.859913 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851506 2586 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 05:44:36.859913 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851509 2586 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 05:44:36.859913 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851511 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 05:44:36.859913 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851514 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 05:44:36.859913 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851517 2586 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 05:44:36.859913 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851520 2586 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 05:44:36.859913 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851523 2586 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 05:44:36.859913 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851525 2586 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 05:44:36.859913 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851528 2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 05:44:36.859913 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851530 2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 05:44:36.859913 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851533 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 05:44:36.859913 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851535 2586 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 05:44:36.859913 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851541 2586 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 05:44:36.859913 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851544 2586 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 05:44:36.859913 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851546 2586 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 05:44:36.860379 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851549 2586 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 05:44:36.860379 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851552 2586 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 05:44:36.860379 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851554 2586 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 05:44:36.860379 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851557 2586 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 05:44:36.860379 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851559 2586 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 05:44:36.860379 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851562 2586 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 05:44:36.860379 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851565 2586 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 05:44:36.860379 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851567 2586 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 05:44:36.860379 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851570 2586 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 05:44:36.860379 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851572 2586 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 05:44:36.860379 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851575 2586 feature_gate.go:386] unrecognized feature gate: Example Nov 01 05:44:36.860379 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851577 2586 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 05:44:36.860379 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851580 2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 05:44:36.860379 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851582 2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 05:44:36.860379 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851584 2586 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 05:44:36.860379 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851587 2586 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 05:44:36.860379 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851590 2586 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 05:44:36.860379 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851592 2586 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 05:44:36.860379 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851595 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 05:44:36.860379 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851597 2586 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 05:44:36.860850 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851599 2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 05:44:36.860850 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851602 2586 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 05:44:36.860850 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851604 2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 05:44:36.860850 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851608 2586 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 05:44:36.860850 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851611 2586 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 05:44:36.860850 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851613 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 05:44:36.860850 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851616 2586 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 05:44:36.860850 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851619 2586 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 05:44:36.860850 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851621 2586 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 05:44:36.860850 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851626 2586 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 05:44:36.860850 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851630 2586 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 05:44:36.860850 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851634 2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 05:44:36.860850 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851636 2586 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 05:44:36.860850 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851639 2586 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 05:44:36.860850 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851642 2586 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 05:44:36.860850 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851645 2586 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 05:44:36.860850 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851648 2586 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 05:44:36.860850 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851651 2586 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 05:44:36.860850 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851653 2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 05:44:36.861339 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851656 2586 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 05:44:36.861339 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851658 2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 05:44:36.861339 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851661 2586 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 05:44:36.861339 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851664 2586 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 05:44:36.861339 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851666 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 05:44:36.861339 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851669 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 05:44:36.861339 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851671 2586 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 05:44:36.861339 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851674 2586 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 05:44:36.861339 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851676 2586 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 05:44:36.861339 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851679 2586 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 05:44:36.861339 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851682 2586 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 05:44:36.861339 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851684 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 05:44:36.861339 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851687 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 05:44:36.861339 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851690 2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 05:44:36.861339 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851694 2586 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 05:44:36.861339 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851696 2586 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 05:44:36.861339 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851699 2586 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 05:44:36.861339 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851702 2586 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 05:44:36.861339 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851704 2586 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 05:44:36.861780 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851707 2586 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 05:44:36.861780 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851709 2586 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 05:44:36.861780 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851712 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 05:44:36.861780 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851716 2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 05:44:36.861780 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851719 2586 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 05:44:36.861780 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851723 2586 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 05:44:36.861780 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851727 2586 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 05:44:36.861780 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:36.851731 2586 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 05:44:36.861780 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.851814 2586 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 05:44:36.861780 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.854009 2586 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 01 05:44:36.861780 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.854944 2586 server.go:1014] "Starting client certificate rotation" Nov 01 05:44:36.861780 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.855047 2586 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 05:44:36.861780 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.855076 2586 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 05:44:36.861780 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.856962 2586 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 05:44:36.861780 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.859790 2586 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 05:44:36.871349 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.871332 2586 log.go:25] "Validated CRI v1 runtime API" Nov 01 05:44:36.876595 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.876576 2586 log.go:25] "Validated CRI v1 image API" Nov 01 05:44:36.877453 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.877424 2586 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 05:44:36.879905 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.879883 2586 fs.go:135] Filesystem UUIDs: map[2f186b44-a1a3-462f-bebc-0e082a2a09a7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d516e640-1b17-4735-bd7e-1b4e7b921c04:/dev/nvme0n1p3] Nov 01 05:44:36.879971 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.879903 2586 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 01 05:44:36.884602 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.884496 2586 manager.go:217] Machine: {Timestamp:2026-11-01 05:44:36.884027587 +0000 UTC m=+0.351146495 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3091431 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec249910224aa2388d41d80fa895cfb4 SystemUUID:ec249910-224a-a238-8d41-d80fa895cfb4 BootID:2f0bdd67-682e-41f4-a088-ce025a98d64e Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4f:1d:06:8d:9d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4f:1d:06:8d:9d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:14:77:84:70:75 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 05:44:36.885274 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.885262 2586 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 05:44:36.885362 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.885350 2586 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 05:44:36.887457 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.887435 2586 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 05:44:36.887590 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.887459 2586 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-195.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 05:44:36.888356 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.888345 2586 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 05:44:36.888388 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.888358 2586 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 05:44:36.888388 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.888371 2586 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 05:44:36.888388 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.888384 2586 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 05:44:36.889630 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.889620 2586 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 05:44:36.894031 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.894018 2586 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 05:44:36.894140 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.894130 2586 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 05:44:36.894872 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.894863 2586 kubelet.go:493] "Attempting to sync node with API server" Nov 01 05:44:36.894906 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.894877 2586 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 05:44:36.894906 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.894891 2586 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 05:44:36.894906 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.894900 2586 kubelet.go:405] "Adding apiserver pod source" Nov 01 05:44:36.894980 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.894909 2586 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 05:44:36.895291 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.895280 2586 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 05:44:36.897669 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.897640 2586 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 05:44:36.904193 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.904180 2586 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 05:44:36.905178 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.905163 2586 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 05:44:36.905215 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.905199 2586 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 05:44:36.905356 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.905347 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 05:44:36.905393 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.905363 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 05:44:36.905393 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.905369 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 05:44:36.905393 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.905375 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 05:44:36.905393 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.905381 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 05:44:36.905393 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.905388 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 05:44:36.906113 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.906104 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 05:44:36.906142 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.906114 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 05:44:36.906142 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.906121 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 05:44:36.906142 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.906126 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 05:44:36.906142 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.906141 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 05:44:36.906264 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.906150 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 05:44:36.907852 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.907842 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 05:44:36.907891 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.907853 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 05:44:36.911677 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.911665 2586 server.go:1263] "Started kubelet" Nov 01 05:44:36.911817 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.911772 2586 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 05:44:36.911859 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.911765 2586 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 05:44:36.911859 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.911852 2586 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 05:44:36.912415 ip-10-0-143-195 systemd[1]: Started Kubernetes Kubelet. Nov 01 05:44:36.913313 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.913288 2586 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 05:44:36.918134 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.918114 2586 server.go:317] "Adding debug handlers to kubelet server" Nov 01 05:44:36.919036 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:36.919008 2586 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 01 05:44:36.923959 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.923936 2586 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 05:44:36.924561 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.924539 2586 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 05:44:36.924678 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:36.924660 2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-195.us-west-1.compute.internal\" not found" Nov 01 05:44:36.925353 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.925336 2586 factory.go:55] Registering systemd factory Nov 01 05:44:36.925353 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.925355 2586 factory.go:223] Registration of the systemd container factory successfully Nov 01 05:44:36.925474 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.925410 2586 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 05:44:36.925474 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.925420 2586 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 05:44:36.925474 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.925438 2586 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 05:44:36.925599 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.925540 2586 reconstruct.go:97] "Volume reconstruction finished" Nov 01 05:44:36.925599 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.925545 2586 reconciler.go:29] "Reconciler: start to sync state" Nov 01 05:44:36.925599 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.925567 2586 factory.go:153] Registering CRI-O factory Nov 01 05:44:36.925599 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.925582 2586 factory.go:223] Registration of the crio container factory successfully Nov 01 05:44:36.925752 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.925630 2586 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 05:44:36.925752 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.925648 2586 factory.go:103] Registering Raw factory Nov 01 05:44:36.925752 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.925666 2586 manager.go:1199] Started watching for new ooms in manager Nov 01 05:44:36.926706 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.926691 2586 manager.go:319] Starting recovery of all containers Nov 01 05:44:36.935960 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.935944 2586 manager.go:324] Recovery completed Nov 01 05:44:36.937491 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:36.937468 2586 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Nov 01 05:44:36.940852 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.940837 2586 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 05:44:36.942425 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.942402 2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-195.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 05:44:36.942500 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.942439 2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-195.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 05:44:36.942500 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.942452 2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-195.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 05:44:36.943025 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.943009 2586 cpu_manager.go:226] "Starting" policy="none" Nov 01 05:44:36.943025 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.943023 2586 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 05:44:36.943153 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.943023 2586 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 05:44:36.943153 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.943057 2586 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 05:44:36.943748 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.943730 2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-195.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 05:44:36.943811 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.943764 2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-195.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 05:44:36.943811 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.943780 2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-195.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 05:44:36.945238 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.945225 2586 policy_none.go:50] "Start" Nov 01 05:44:36.945282 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.945242 2586 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 05:44:36.945282 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.945251 2586 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 05:44:36.947212 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.947202 2586 policy_none.go:44] "Start" Nov 01 05:44:36.988595 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.988561 2586 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 05:44:37.012550 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.993819 2586 manager.go:341] "Starting Device Plugin manager" Nov 01 05:44:37.012550 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:36.993852 2586 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 05:44:37.012550 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.993863 2586 server.go:87] "Starting device plugin registration server" Nov 01 05:44:37.012550 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.994146 2586 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 05:44:37.012550 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.994159 2586 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 05:44:37.012550 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.994257 2586 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 05:44:37.012550 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.994330 2586 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 05:44:37.012550 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:36.994337 2586 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 05:44:37.012550 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:36.994852 2586 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 05:44:37.012550 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:36.994903 2586 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-195.us-west-1.compute.internal\" not found" Nov 01 05:44:37.012550 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.004603 2586 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 05:44:37.012550 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.005938 2586 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 05:44:37.012550 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.005956 2586 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 05:44:37.012550 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.005974 2586 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 05:44:37.012550 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:37.006025 2586 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 01 05:44:37.061492 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:37.061238 2586 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-195.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 01 05:44:37.061492 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.061461 2586 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-195.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 01 05:44:37.062315 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:37.061255 2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-195.us-west-1.compute.internal.18e3d58cb3c31594 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-195.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-195.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 05:44:36.911633812 +0000 UTC m=+0.378752719,LastTimestamp:2026-11-01 05:44:36.911633812 +0000 UTC m=+0.378752719,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-195.us-west-1.compute.internal,}" Nov 01 05:44:37.094543 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.094498 2586 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 05:44:37.095390 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.095374 2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-195.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 05:44:37.095465 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.095403 2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-195.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 05:44:37.095465 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.095417 2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-195.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 05:44:37.095465 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.095442 2586 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:44:37.107087 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.107064 2586 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-195.us-west-1.compute.internal"] Nov 01 05:44:37.107148 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.107142 2586 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 05:44:37.107880 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.107866 2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-195.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 05:44:37.107932 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.107893 2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-195.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 05:44:37.107932 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.107905 2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-195.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 05:44:37.109963 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.109948 2586 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 05:44:37.110137 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.110122 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:44:37.110189 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.110149 2586 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 05:44:37.110663 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.110646 2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-195.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 05:44:37.110758 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.110680 2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-195.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 05:44:37.110758 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.110696 2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-195.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 05:44:37.110758 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.110651 2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-195.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 05:44:37.110758 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.110751 2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-195.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 05:44:37.110952 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.110764 2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-195.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 05:44:37.112864 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.112847 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:44:37.112936 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.112877 2586 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 05:44:37.113650 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.113632 2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-195.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 05:44:37.113727 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.113664 2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-195.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 05:44:37.113727 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.113680 2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-195.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 05:44:37.127161 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.127142 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ac965f1cfbc1129016d728e686326ed6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal\" (UID: \"ac965f1cfbc1129016d728e686326ed6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:44:37.127247 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.127167 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac965f1cfbc1129016d728e686326ed6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal\" (UID: \"ac965f1cfbc1129016d728e686326ed6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:44:37.127247 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.127184 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c656c4baa8463ac8d24886e04ca99821-config\") pod \"kube-apiserver-proxy-ip-10-0-143-195.us-west-1.compute.internal\" (UID: \"c656c4baa8463ac8d24886e04ca99821\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:44:37.138086 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.138064 2586 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 05:44:37.138163 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.138118 2586 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q5wgc" Nov 01 05:44:37.143887 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:37.143869 2586 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-195.us-west-1.compute.internal\" not found" node="ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:44:37.146427 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:37.146410 2586 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-195.us-west-1.compute.internal\" not found" node="ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:44:37.170713 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:37.170686 2586 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Nov 01 05:44:37.170713 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:37.170697 2586 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 01 05:44:37.170930 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:37.170872 2586 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-195.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 01 05:44:37.170976 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:37.170939 2586 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 01 05:44:37.171055 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:37.170954 2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-195.us-west-1.compute.internal.18e3d58cb598e3b3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-195.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-195.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-195.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 05:44:36.942422963 +0000 UTC m=+0.409541871,LastTimestamp:2026-11-01 05:44:36.942422963 +0000 UTC m=+0.409541871,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-195.us-west-1.compute.internal,}" Nov 01 05:44:37.226074 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:37.226052 2586 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-195.us-west-1.compute.internal\" not found" node="ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:44:37.228198 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.228183 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ac965f1cfbc1129016d728e686326ed6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal\" (UID: \"ac965f1cfbc1129016d728e686326ed6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:44:37.228265 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.228207 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac965f1cfbc1129016d728e686326ed6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal\" (UID: \"ac965f1cfbc1129016d728e686326ed6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:44:37.228265 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.228223 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c656c4baa8463ac8d24886e04ca99821-config\") pod \"kube-apiserver-proxy-ip-10-0-143-195.us-west-1.compute.internal\" (UID: \"c656c4baa8463ac8d24886e04ca99821\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:44:37.228336 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.228282 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c656c4baa8463ac8d24886e04ca99821-config\") pod \"kube-apiserver-proxy-ip-10-0-143-195.us-west-1.compute.internal\" (UID: \"c656c4baa8463ac8d24886e04ca99821\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:44:37.228336 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.228302 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ac965f1cfbc1129016d728e686326ed6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal\" (UID: \"ac965f1cfbc1129016d728e686326ed6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:44:37.228336 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.228310 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac965f1cfbc1129016d728e686326ed6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal\" (UID: \"ac965f1cfbc1129016d728e686326ed6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:44:37.365571 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:37.365492 2586 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-143-195.us-west-1.compute.internal\" not found" node="ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:44:37.426769 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.426743 2586 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 05:44:37.427808 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.427789 2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-195.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 05:44:37.427913 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.427822 2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-195.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 05:44:37.427913 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.427836 2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-195.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 05:44:37.427913 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.427864 2586 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:44:37.449054 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.449038 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:44:37.451876 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.451855 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:44:37.483491 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:37.483472 2586 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:44:37.483575 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:37.483495 2586 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-195.us-west-1.compute.internal\": node \"ip-10-0-143-195.us-west-1.compute.internal\" not found" Nov 01 05:44:37.593959 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:37.593932 2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-195.us-west-1.compute.internal\" not found" Nov 01 05:44:38.860092 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:38.860064 2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-195.us-west-1.compute.internal\" not found" Nov 01 05:44:38.961156 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:38.961128 2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-195.us-west-1.compute.internal\" not found" Nov 01 05:44:39.020532 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:39.020504 2586 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 01 05:44:39.020968 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:39.020646 2586 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-143-195.us-west-1.compute.internal.18e3d58cb5993d19\": read tcp 10.0.143.195:34564->18.189.164.78:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-143-195.us-west-1.compute.internal.18e3d58cb5993d19 default 4797 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-195.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-195.us-west-1.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-195.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 05:44:36 +0000 UTC,LastTimestamp:2026-11-01 05:44:37.107900124 +0000 UTC m=+0.575019033,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-195.us-west-1.compute.internal,}" Nov 01 05:44:39.061857 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:39.061832 2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-195.us-west-1.compute.internal\" not found" Nov 01 05:44:39.162579 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:39.162520 2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-195.us-west-1.compute.internal\" not found" Nov 01 05:44:39.263573 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:39.263543 2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-195.us-west-1.compute.internal\" not found" Nov 01 05:44:39.308971 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:39.308926 2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 05:39:38 +0000 UTC" deadline="2028-07-08 18:21:06.855790144 +0000 UTC" Nov 01 05:44:39.308971 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:39.308960 2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14772h36m27.54683351s" Nov 01 05:44:39.364539 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:39.364518 2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-195.us-west-1.compute.internal\" not found" Nov 01 05:44:39.371672 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:39.371655 2586 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 05:44:39.372705 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:39.372687 2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-195.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 05:44:39.372791 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:39.372722 2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-195.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 05:44:39.372791 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:39.372734 2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-195.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 05:44:39.465004 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:39.464935 2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-195.us-west-1.compute.internal\" not found" Nov 01 05:44:39.502338 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:39.502315 2586 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 05:44:39.565839 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:39.565811 2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-195.us-west-1.compute.internal\" not found" Nov 01 05:44:39.666268 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:39.666239 2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-195.us-west-1.compute.internal\" not found" Nov 01 05:44:39.744674 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:39.744655 2586 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 05:44:39.755665 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:39.755637 2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc656c4baa8463ac8d24886e04ca99821.slice/crio-9fd52c62387ec79eddc2fc1c21b7174cd35bdb356617695d27af2b5a992b9442 WatchSource:0}: Error finding container 9fd52c62387ec79eddc2fc1c21b7174cd35bdb356617695d27af2b5a992b9442: Status 404 returned error can't find the container with id 9fd52c62387ec79eddc2fc1c21b7174cd35bdb356617695d27af2b5a992b9442 Nov 01 05:44:39.756000 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:39.755979 2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac965f1cfbc1129016d728e686326ed6.slice/crio-18a61660adc8698ee2d89530b878234ff5b08430b0da524a730460af2bbd2fb0 WatchSource:0}: Error finding container 18a61660adc8698ee2d89530b878234ff5b08430b0da524a730460af2bbd2fb0: Status 404 returned error can't find the container with id 18a61660adc8698ee2d89530b878234ff5b08430b0da524a730460af2bbd2fb0 Nov 01 05:44:39.761706 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:39.761691 2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 05:44:39.779911 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:39.779892 2586 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 05:44:39.791011 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:39.790993 2586 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:44:39.853924 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:39.853900 2586 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 05:44:39.855820 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:39.855806 2586 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:44:39.915443 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:39.915419 2586 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 05:44:39.915585 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:39.915461 2586 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 05:44:40.061056 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.060962 2586 apiserver.go:52] "Watching apiserver" Nov 01 05:44:40.091028 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.091007 2586 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 01 05:44:40.163759 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.163724 2586 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 05:44:40.164013 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.163997 2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-143-195.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x","openshift-image-registry/node-ca-tw96g","openshift-multus/multus-additional-cni-plugins-b7rrq","openshift-multus/network-metrics-daemon-tt8gq","openshift-multus/multus-qh47z","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal","openshift-network-diagnostics/network-check-target-ljcjx","kube-system/konnectivity-agent-n7mks","openshift-cluster-node-tuning-operator/tuned-cb599","openshift-dns/node-resolver-2qtwx","openshift-network-operator/iptables-alerter-qbg2s","openshift-ovn-kubernetes/ovnkube-node-7bgdz","kube-system/global-pull-secret-syncer-r5l82"] Nov 01 05:44:40.167716 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.167700 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.169782 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.169763 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.171888 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.171870 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tw96g" Nov 01 05:44:40.174051 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.174029 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:44:40.176238 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.176216 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:44:40.176333 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:40.176310 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tt8gq" podUID="a1f5a77d-43d5-48bc-a8b5-5db8f1152853" Nov 01 05:44:40.178454 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.178436 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.180755 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.180739 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n7mks" Nov 01 05:44:40.182949 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.182934 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:44:40.183034 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:40.183003 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5l82" podUID="7dc7486d-3188-4bb8-b907-b653caa18660" Nov 01 05:44:40.185387 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.185368 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:44:40.187509 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.187492 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljcjx" Nov 01 05:44:40.187596 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:40.187550 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljcjx" podUID="529e2d97-b167-4011-a215-02ece8519c00" Nov 01 05:44:40.189817 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.189801 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qbg2s" Nov 01 05:44:40.192029 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.191994 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal" event={"ID":"ac965f1cfbc1129016d728e686326ed6","Type":"ContainerStarted","Data":"18a61660adc8698ee2d89530b878234ff5b08430b0da524a730460af2bbd2fb0"} Nov 01 05:44:40.192148 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.192034 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-195.us-west-1.compute.internal" event={"ID":"c656c4baa8463ac8d24886e04ca99821","Type":"ContainerStarted","Data":"9fd52c62387ec79eddc2fc1c21b7174cd35bdb356617695d27af2b5a992b9442"} Nov 01 05:44:40.192148 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.192036 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2qtwx" Nov 01 05:44:40.205497 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.205452 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-hostroot\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.205587 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.205514 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/197fb18c-6703-4deb-93b3-0186d7ceed67-agent-certs\") pod \"konnectivity-agent-n7mks\" (UID: \"197fb18c-6703-4deb-93b3-0186d7ceed67\") " pod="kube-system/konnectivity-agent-n7mks" Nov 01 05:44:40.205587 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.205549 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2xnh\" (UniqueName: \"kubernetes.io/projected/529e2d97-b167-4011-a215-02ece8519c00-kube-api-access-k2xnh\") pod \"network-check-target-ljcjx\" (UID: \"529e2d97-b167-4011-a215-02ece8519c00\") " pod="openshift-network-diagnostics/network-check-target-ljcjx" Nov 01 05:44:40.205587 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.205577 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0719dc90-dbae-425a-8439-812425e1b54c-host-slash\") pod \"iptables-alerter-qbg2s\" (UID: \"0719dc90-dbae-425a-8439-812425e1b54c\") " pod="openshift-network-operator/iptables-alerter-qbg2s" Nov 01 05:44:40.205691 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.205648 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrtw2\" (UniqueName: \"kubernetes.io/projected/ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8-kube-api-access-rrtw2\") pod \"node-resolver-2qtwx\" (UID: \"ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8\") " pod="openshift-dns/node-resolver-2qtwx" Nov 01 05:44:40.205727 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.205702 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-etc-sysctl-d\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.205761 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.205730 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-run-ovn\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.205761 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.205749 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/21519345-86b8-441e-a395-1e8385cd3732-ovn-node-metrics-cert\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.205817 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.205764 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-system-cni-dir\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.205859 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.205803 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-host-run-k8s-cni-cncf-io\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.205899 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.205874 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-slash\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.205942 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.205911 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6xv7\" (UniqueName: \"kubernetes.io/projected/21519345-86b8-441e-a395-1e8385cd3732-kube-api-access-g6xv7\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.205976 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.205933 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-host-var-lib-cni-multus\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.206306 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.206285 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-etc-openvswitch\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.206394 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.206345 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-host-var-lib-cni-bin\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.206454 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.206395 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz6tw\" (UniqueName: \"kubernetes.io/projected/6cd67a78-09f1-4988-99b0-1ac9097f2f39-kube-api-access-kz6tw\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:44:40.206519 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.206448 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-etc-kubernetes\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.206556 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.206528 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-var-lib-openvswitch\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.206602 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.206589 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-cni-netd\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.206644 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.206620 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzxml\" (UniqueName: \"kubernetes.io/projected/f5705a40-d3e4-40a8-a561-17c435f388b1-kube-api-access-dzxml\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:44:40.206644 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.206638 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/67ae987f-79b9-4852-98bd-e0537c412b17-multus-daemon-config\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.206744 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.206724 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/197fb18c-6703-4deb-93b3-0186d7ceed67-konnectivity-ca\") pod \"konnectivity-agent-n7mks\" (UID: \"197fb18c-6703-4deb-93b3-0186d7ceed67\") " pod="kube-system/konnectivity-agent-n7mks" Nov 01 05:44:40.206796 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.206772 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-etc-modprobe-d\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.206841 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.206803 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s56k\" (UniqueName: \"kubernetes.io/projected/2acbc647-2361-4935-b503-5bd9af74f822-kube-api-access-4s56k\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.206881 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.206858 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cde21439-a494-47cd-8cc5-161fc96a6701-host\") pod \"node-ca-tw96g\" (UID: \"cde21439-a494-47cd-8cc5-161fc96a6701\") " pod="openshift-image-registry/node-ca-tw96g" Nov 01 05:44:40.206881 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.206875 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5705a40-d3e4-40a8-a561-17c435f388b1-system-cni-dir\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:44:40.206963 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.206888 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6cd67a78-09f1-4988-99b0-1ac9097f2f39-device-dir\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:44:40.206963 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.206901 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-run-netns\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.206963 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.206914 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/21519345-86b8-441e-a395-1e8385cd3732-env-overrides\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.206963 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.206950 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dbh2\" (UniqueName: \"kubernetes.io/projected/cde21439-a494-47cd-8cc5-161fc96a6701-kube-api-access-8dbh2\") pod \"node-ca-tw96g\" (UID: \"cde21439-a494-47cd-8cc5-161fc96a6701\") " pod="openshift-image-registry/node-ca-tw96g" Nov 01 05:44:40.207100 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.206984 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gbgq\" (UniqueName: \"kubernetes.io/projected/67ae987f-79b9-4852-98bd-e0537c412b17-kube-api-access-7gbgq\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.207100 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.207007 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cde21439-a494-47cd-8cc5-161fc96a6701-serviceca\") pod \"node-ca-tw96g\" (UID: \"cde21439-a494-47cd-8cc5-161fc96a6701\") " pod="openshift-image-registry/node-ca-tw96g" Nov 01 05:44:40.207100 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.207035 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7dc7486d-3188-4bb8-b907-b653caa18660-kubelet-config\") pod \"global-pull-secret-syncer-r5l82\" (UID: \"7dc7486d-3188-4bb8-b907-b653caa18660\") " pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:44:40.207100 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.207067 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2acbc647-2361-4935-b503-5bd9af74f822-tmp\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.207100 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.207091 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-node-log\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.207272 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.207106 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2acbc647-2361-4935-b503-5bd9af74f822-etc-tuned\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.207272 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.207127 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-kubelet\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.207272 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.207163 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/21519345-86b8-441e-a395-1e8385cd3732-ovnkube-script-lib\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.207272 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.207184 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8fz2\" (UniqueName: \"kubernetes.io/projected/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-kube-api-access-f8fz2\") pod \"network-metrics-daemon-tt8gq\" (UID: \"a1f5a77d-43d5-48bc-a8b5-5db8f1152853\") " pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:44:40.207272 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.207199 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-cnibin\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.207272 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.207225 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-host-var-lib-kubelet\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.207272 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.207253 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6cd67a78-09f1-4988-99b0-1ac9097f2f39-etc-selinux\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:44:40.207513 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.207305 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-lib-modules\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.207513 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.207331 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5705a40-d3e4-40a8-a561-17c435f388b1-cnibin\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:44:40.207513 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.207376 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f5705a40-d3e4-40a8-a561-17c435f388b1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:44:40.207513 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.207406 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6cd67a78-09f1-4988-99b0-1ac9097f2f39-kubelet-dir\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:44:40.207513 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.207425 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6cd67a78-09f1-4988-99b0-1ac9097f2f39-sys-fs\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:44:40.207513 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.207487 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-etc-sysctl-conf\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.207698 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.207518 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-var-lib-kubelet\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.207698 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.207534 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-systemd-units\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.207698 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.207550 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-run-systemd\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.207698 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.207580 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-run-ovn-kubernetes\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.207698 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.207595 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7dc7486d-3188-4bb8-b907-b653caa18660-dbus\") pod \"global-pull-secret-syncer-r5l82\" (UID: \"7dc7486d-3188-4bb8-b907-b653caa18660\") " pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:44:40.207698 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.207617 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7dc7486d-3188-4bb8-b907-b653caa18660-original-pull-secret\") pod \"global-pull-secret-syncer-r5l82\" (UID: \"7dc7486d-3188-4bb8-b907-b653caa18660\") " pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:44:40.207698 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.207646 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0719dc90-dbae-425a-8439-812425e1b54c-iptables-alerter-script\") pod \"iptables-alerter-qbg2s\" (UID: \"0719dc90-dbae-425a-8439-812425e1b54c\") " pod="openshift-network-operator/iptables-alerter-qbg2s" Nov 01 05:44:40.207698 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.207697 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-run-openvswitch\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.207928 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.207713 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/21519345-86b8-441e-a395-1e8385cd3732-ovnkube-config\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.207928 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.207727 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5705a40-d3e4-40a8-a561-17c435f388b1-cni-binary-copy\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:44:40.207928 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.207740 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6cd67a78-09f1-4988-99b0-1ac9097f2f39-registration-dir\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:44:40.207928 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.207753 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-cni-bin\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.207928 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.207800 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-multus-cni-dir\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.207928 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.207834 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-multus-conf-dir\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.207928 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.207860 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-host-run-multus-certs\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.207928 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.207903 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-sys\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.207928 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.207928 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-log-socket\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.208260 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.207959 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5705a40-d3e4-40a8-a561-17c435f388b1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:44:40.208260 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.207977 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f5705a40-d3e4-40a8-a561-17c435f388b1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:44:40.208260 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.208024 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs\") pod \"network-metrics-daemon-tt8gq\" (UID: \"a1f5a77d-43d5-48bc-a8b5-5db8f1152853\") " pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:44:40.208260 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.208061 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-host-run-netns\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.208260 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.208079 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6cd67a78-09f1-4988-99b0-1ac9097f2f39-socket-dir\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:44:40.208260 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.208129 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdrd9\" (UniqueName: \"kubernetes.io/projected/0719dc90-dbae-425a-8439-812425e1b54c-kube-api-access-mdrd9\") pod \"iptables-alerter-qbg2s\" (UID: \"0719dc90-dbae-425a-8439-812425e1b54c\") " pod="openshift-network-operator/iptables-alerter-qbg2s" Nov 01 05:44:40.208260 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.208168 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-etc-kubernetes\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.208260 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.208200 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-etc-systemd\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.208260 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.208221 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-run\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.208260 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.208243 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-host\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.208662 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.208291 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67ae987f-79b9-4852-98bd-e0537c412b17-cni-binary-copy\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.208662 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.208339 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8-hosts-file\") pod \"node-resolver-2qtwx\" (UID: \"ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8\") " pod="openshift-dns/node-resolver-2qtwx" Nov 01 05:44:40.208662 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.208364 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8-tmp-dir\") pod \"node-resolver-2qtwx\" (UID: \"ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8\") " pod="openshift-dns/node-resolver-2qtwx" Nov 01 05:44:40.208662 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.208392 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-etc-sysconfig\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.208662 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.208426 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.208662 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.208463 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-multus-socket-dir-parent\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.220450 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.220430 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 05:44:40.220548 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.220525 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rpsgx\"" Nov 01 05:44:40.220592 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.220559 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 05:44:40.221487 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.221456 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 05:44:40.222021 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.222001 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 05:44:40.222097 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.222005 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 05:44:40.222134 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.222094 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 05:44:40.222134 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.222122 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 05:44:40.222269 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.222258 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vbc6d\"" Nov 01 05:44:40.222322 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.222309 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 05:44:40.223371 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.223356 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 05:44:40.223803 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.223788 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 05:44:40.223844 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.223789 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9tjqf\"" Nov 01 05:44:40.223844 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.223823 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 05:44:40.225689 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.225676 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 05:44:40.225950 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.225938 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 05:44:40.226120 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.226109 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 05:44:40.226189 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.226178 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 05:44:40.226243 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.226217 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-m2d6d\"" Nov 01 05:44:40.226280 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.226240 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 05:44:40.230244 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.230230 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rz4ht\"" Nov 01 05:44:40.230304 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.230250 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 05:44:40.232542 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.232526 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 05:44:40.232617 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.232579 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 05:44:40.264956 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.264934 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 05:44:40.265420 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.265398 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 05:44:40.265420 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.265407 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 05:44:40.265562 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.265451 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 05:44:40.265562 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.265494 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 05:44:40.269715 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.269699 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qfzvw\"" Nov 01 05:44:40.269790 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.269693 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 05:44:40.269901 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.269891 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-695z8\"" Nov 01 05:44:40.270013 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.269998 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 05:44:40.270064 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.270000 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 05:44:40.270064 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.270057 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jzphk\"" Nov 01 05:44:40.270256 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.270242 2586 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 05:44:40.270294 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.270248 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xlcv8\"" Nov 01 05:44:40.270294 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.270281 2586 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zl9mv" Nov 01 05:44:40.291946 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.291926 2586 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 05:44:40.308789 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.308768 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f5705a40-d3e4-40a8-a561-17c435f388b1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:44:40.308883 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.308801 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6cd67a78-09f1-4988-99b0-1ac9097f2f39-kubelet-dir\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:44:40.308883 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.308825 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6cd67a78-09f1-4988-99b0-1ac9097f2f39-sys-fs\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:44:40.308883 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.308847 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-etc-sysctl-conf\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.308883 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.308870 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-var-lib-kubelet\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.309062 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.308892 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-systemd-units\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.309062 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.308908 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6cd67a78-09f1-4988-99b0-1ac9097f2f39-sys-fs\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:44:40.309062 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.308922 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-run-systemd\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.309062 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.308943 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-systemd-units\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.309062 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.308909 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6cd67a78-09f1-4988-99b0-1ac9097f2f39-kubelet-dir\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:44:40.309062 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.308945 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-var-lib-kubelet\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.309062 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.308958 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-run-ovn-kubernetes\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.309062 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.308988 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-run-systemd\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.309062 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.308988 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-run-ovn-kubernetes\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.309062 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309000 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-etc-sysctl-conf\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.309062 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309003 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7dc7486d-3188-4bb8-b907-b653caa18660-dbus\") pod \"global-pull-secret-syncer-r5l82\" (UID: \"7dc7486d-3188-4bb8-b907-b653caa18660\") " pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:44:40.309062 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309047 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7dc7486d-3188-4bb8-b907-b653caa18660-original-pull-secret\") pod \"global-pull-secret-syncer-r5l82\" (UID: \"7dc7486d-3188-4bb8-b907-b653caa18660\") " pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:44:40.309641 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309075 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0719dc90-dbae-425a-8439-812425e1b54c-iptables-alerter-script\") pod \"iptables-alerter-qbg2s\" (UID: \"0719dc90-dbae-425a-8439-812425e1b54c\") " pod="openshift-network-operator/iptables-alerter-qbg2s" Nov 01 05:44:40.309641 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309103 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-run-openvswitch\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.309641 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309126 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/21519345-86b8-441e-a395-1e8385cd3732-ovnkube-config\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.309641 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309148 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5705a40-d3e4-40a8-a561-17c435f388b1-cni-binary-copy\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:44:40.309641 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:40.309165 2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 05:44:40.309641 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309170 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6cd67a78-09f1-4988-99b0-1ac9097f2f39-registration-dir\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:44:40.309641 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309174 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-run-openvswitch\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.309641 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309185 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7dc7486d-3188-4bb8-b907-b653caa18660-dbus\") pod \"global-pull-secret-syncer-r5l82\" (UID: \"7dc7486d-3188-4bb8-b907-b653caa18660\") " pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:44:40.309641 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309193 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-cni-bin\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.309641 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:40.309240 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dc7486d-3188-4bb8-b907-b653caa18660-original-pull-secret podName:7dc7486d-3188-4bb8-b907-b653caa18660 nodeName:}" failed. No retries permitted until 2026-11-01 05:44:40.809218465 +0000 UTC m=+3.110844072 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7dc7486d-3188-4bb8-b907-b653caa18660-original-pull-secret") pod "global-pull-secret-syncer-r5l82" (UID: "7dc7486d-3188-4bb8-b907-b653caa18660") : object "kube-system"/"original-pull-secret" not registered Nov 01 05:44:40.309641 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309237 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6cd67a78-09f1-4988-99b0-1ac9097f2f39-registration-dir\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:44:40.309641 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309244 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-cni-bin\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.309641 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309257 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-multus-cni-dir\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.309641 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309282 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-multus-conf-dir\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.309641 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309305 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-host-run-multus-certs\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.309641 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309325 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-sys\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.309641 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309349 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-log-socket\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.310450 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309359 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-multus-conf-dir\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.310450 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309360 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-host-run-multus-certs\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.310450 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309373 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5705a40-d3e4-40a8-a561-17c435f388b1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:44:40.310450 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309403 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-sys\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.310450 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309411 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f5705a40-d3e4-40a8-a561-17c435f388b1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:44:40.310450 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309414 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-log-socket\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.310450 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309414 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f5705a40-d3e4-40a8-a561-17c435f388b1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:44:40.310450 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309437 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-multus-cni-dir\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.310450 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309437 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs\") pod \"network-metrics-daemon-tt8gq\" (UID: \"a1f5a77d-43d5-48bc-a8b5-5db8f1152853\") " pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:44:40.310450 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309503 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-host-run-netns\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.310450 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:40.309531 2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 05:44:40.310450 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309574 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5705a40-d3e4-40a8-a561-17c435f388b1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:44:40.310450 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309584 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-host-run-netns\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.310450 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309531 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6cd67a78-09f1-4988-99b0-1ac9097f2f39-socket-dir\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:44:40.310450 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:40.309590 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs podName:a1f5a77d-43d5-48bc-a8b5-5db8f1152853 nodeName:}" failed. No retries permitted until 2026-11-01 05:44:40.80957441 +0000 UTC m=+3.111200016 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs") pod "network-metrics-daemon-tt8gq" (UID: "a1f5a77d-43d5-48bc-a8b5-5db8f1152853") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 05:44:40.310450 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309631 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdrd9\" (UniqueName: \"kubernetes.io/projected/0719dc90-dbae-425a-8439-812425e1b54c-kube-api-access-mdrd9\") pod \"iptables-alerter-qbg2s\" (UID: \"0719dc90-dbae-425a-8439-812425e1b54c\") " pod="openshift-network-operator/iptables-alerter-qbg2s" Nov 01 05:44:40.310450 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309653 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-etc-kubernetes\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.311178 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309674 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-etc-systemd\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.311178 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309697 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-run\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.311178 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309704 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5705a40-d3e4-40a8-a561-17c435f388b1-cni-binary-copy\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:44:40.311178 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309712 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-etc-systemd\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.311178 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309718 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-host\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.311178 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309755 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0719dc90-dbae-425a-8439-812425e1b54c-iptables-alerter-script\") pod \"iptables-alerter-qbg2s\" (UID: \"0719dc90-dbae-425a-8439-812425e1b54c\") " pod="openshift-network-operator/iptables-alerter-qbg2s" Nov 01 05:44:40.311178 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309767 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-host\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.311178 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309763 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-run\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.311178 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309763 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-etc-kubernetes\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.311178 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309794 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67ae987f-79b9-4852-98bd-e0537c412b17-cni-binary-copy\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.311178 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309804 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/21519345-86b8-441e-a395-1e8385cd3732-ovnkube-config\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.311178 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309818 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8-hosts-file\") pod \"node-resolver-2qtwx\" (UID: \"ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8\") " pod="openshift-dns/node-resolver-2qtwx" Nov 01 05:44:40.311178 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309843 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8-tmp-dir\") pod \"node-resolver-2qtwx\" (UID: \"ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8\") " pod="openshift-dns/node-resolver-2qtwx" Nov 01 05:44:40.311178 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309867 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-etc-sysconfig\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.311178 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309891 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.311178 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309900 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8-hosts-file\") pod \"node-resolver-2qtwx\" (UID: \"ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8\") " pod="openshift-dns/node-resolver-2qtwx" Nov 01 05:44:40.311178 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309908 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f5705a40-d3e4-40a8-a561-17c435f388b1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:44:40.311178 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309918 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-multus-socket-dir-parent\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.311918 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309941 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-etc-sysconfig\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.311918 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309940 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-hostroot\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.311918 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309969 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/197fb18c-6703-4deb-93b3-0186d7ceed67-agent-certs\") pod \"konnectivity-agent-n7mks\" (UID: \"197fb18c-6703-4deb-93b3-0186d7ceed67\") " pod="kube-system/konnectivity-agent-n7mks" Nov 01 05:44:40.311918 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309967 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.311918 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309976 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-hostroot\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.311918 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.309994 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2xnh\" (UniqueName: \"kubernetes.io/projected/529e2d97-b167-4011-a215-02ece8519c00-kube-api-access-k2xnh\") pod \"network-check-target-ljcjx\" (UID: \"529e2d97-b167-4011-a215-02ece8519c00\") " pod="openshift-network-diagnostics/network-check-target-ljcjx" Nov 01 05:44:40.311918 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310013 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0719dc90-dbae-425a-8439-812425e1b54c-host-slash\") pod \"iptables-alerter-qbg2s\" (UID: \"0719dc90-dbae-425a-8439-812425e1b54c\") " pod="openshift-network-operator/iptables-alerter-qbg2s" Nov 01 05:44:40.311918 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310028 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-multus-socket-dir-parent\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.311918 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310036 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrtw2\" (UniqueName: \"kubernetes.io/projected/ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8-kube-api-access-rrtw2\") pod \"node-resolver-2qtwx\" (UID: \"ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8\") " pod="openshift-dns/node-resolver-2qtwx" Nov 01 05:44:40.311918 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310053 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-etc-sysctl-d\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.311918 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310065 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0719dc90-dbae-425a-8439-812425e1b54c-host-slash\") pod \"iptables-alerter-qbg2s\" (UID: \"0719dc90-dbae-425a-8439-812425e1b54c\") " pod="openshift-network-operator/iptables-alerter-qbg2s" Nov 01 05:44:40.311918 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310073 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-run-ovn\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.311918 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310136 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8-tmp-dir\") pod \"node-resolver-2qtwx\" (UID: \"ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8\") " pod="openshift-dns/node-resolver-2qtwx" Nov 01 05:44:40.311918 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310147 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/21519345-86b8-441e-a395-1e8385cd3732-ovn-node-metrics-cert\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.311918 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310172 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6cd67a78-09f1-4988-99b0-1ac9097f2f39-socket-dir\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:44:40.311918 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310180 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-system-cni-dir\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.311918 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310203 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-run-ovn\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.311918 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310230 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-system-cni-dir\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.312777 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310260 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-host-run-k8s-cni-cncf-io\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.312777 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310214 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-host-run-k8s-cni-cncf-io\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.312777 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310276 2586 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 05:44:40.312777 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310290 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-slash\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.312777 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310306 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-etc-sysctl-d\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.312777 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310323 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6xv7\" (UniqueName: \"kubernetes.io/projected/21519345-86b8-441e-a395-1e8385cd3732-kube-api-access-g6xv7\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.312777 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310339 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-host-var-lib-cni-multus\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.312777 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310339 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67ae987f-79b9-4852-98bd-e0537c412b17-cni-binary-copy\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.312777 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310350 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-slash\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.312777 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310355 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-etc-openvswitch\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.312777 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310379 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-etc-openvswitch\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.312777 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310384 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-host-var-lib-cni-bin\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.312777 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310393 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-host-var-lib-cni-multus\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.312777 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310407 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-host-var-lib-cni-bin\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.312777 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310412 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz6tw\" (UniqueName: \"kubernetes.io/projected/6cd67a78-09f1-4988-99b0-1ac9097f2f39-kube-api-access-kz6tw\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:44:40.312777 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310435 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-etc-kubernetes\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.312777 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310450 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-var-lib-openvswitch\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.312777 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310469 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-cni-netd\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.313595 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310510 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-var-lib-openvswitch\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.313595 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310511 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzxml\" (UniqueName: \"kubernetes.io/projected/f5705a40-d3e4-40a8-a561-17c435f388b1-kube-api-access-dzxml\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:44:40.313595 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310561 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-etc-kubernetes\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.313595 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310582 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/67ae987f-79b9-4852-98bd-e0537c412b17-multus-daemon-config\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.313595 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310605 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/197fb18c-6703-4deb-93b3-0186d7ceed67-konnectivity-ca\") pod \"konnectivity-agent-n7mks\" (UID: \"197fb18c-6703-4deb-93b3-0186d7ceed67\") " pod="kube-system/konnectivity-agent-n7mks" Nov 01 05:44:40.313595 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310628 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-etc-modprobe-d\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.313595 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310658 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-cni-netd\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.313595 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310659 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s56k\" (UniqueName: \"kubernetes.io/projected/2acbc647-2361-4935-b503-5bd9af74f822-kube-api-access-4s56k\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.313595 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310688 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cde21439-a494-47cd-8cc5-161fc96a6701-host\") pod \"node-ca-tw96g\" (UID: \"cde21439-a494-47cd-8cc5-161fc96a6701\") " pod="openshift-image-registry/node-ca-tw96g" Nov 01 05:44:40.313595 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310704 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5705a40-d3e4-40a8-a561-17c435f388b1-system-cni-dir\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:44:40.313595 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310737 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6cd67a78-09f1-4988-99b0-1ac9097f2f39-device-dir\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:44:40.313595 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310761 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-run-netns\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.313595 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310790 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/21519345-86b8-441e-a395-1e8385cd3732-env-overrides\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.313595 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310815 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dbh2\" (UniqueName: \"kubernetes.io/projected/cde21439-a494-47cd-8cc5-161fc96a6701-kube-api-access-8dbh2\") pod \"node-ca-tw96g\" (UID: \"cde21439-a494-47cd-8cc5-161fc96a6701\") " pod="openshift-image-registry/node-ca-tw96g" Nov 01 05:44:40.313595 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310839 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gbgq\" (UniqueName: \"kubernetes.io/projected/67ae987f-79b9-4852-98bd-e0537c412b17-kube-api-access-7gbgq\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.313595 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310862 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cde21439-a494-47cd-8cc5-161fc96a6701-serviceca\") pod \"node-ca-tw96g\" (UID: \"cde21439-a494-47cd-8cc5-161fc96a6701\") " pod="openshift-image-registry/node-ca-tw96g" Nov 01 05:44:40.313595 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310865 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6cd67a78-09f1-4988-99b0-1ac9097f2f39-device-dir\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:44:40.314338 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310889 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7dc7486d-3188-4bb8-b907-b653caa18660-kubelet-config\") pod \"global-pull-secret-syncer-r5l82\" (UID: \"7dc7486d-3188-4bb8-b907-b653caa18660\") " pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:44:40.314338 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310911 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cde21439-a494-47cd-8cc5-161fc96a6701-host\") pod \"node-ca-tw96g\" (UID: \"cde21439-a494-47cd-8cc5-161fc96a6701\") " pod="openshift-image-registry/node-ca-tw96g" Nov 01 05:44:40.314338 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310914 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2acbc647-2361-4935-b503-5bd9af74f822-tmp\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.314338 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310950 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-node-log\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.314338 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310975 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2acbc647-2361-4935-b503-5bd9af74f822-etc-tuned\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.314338 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.310977 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-run-netns\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.314338 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.311003 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-kubelet\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.314338 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.311023 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/21519345-86b8-441e-a395-1e8385cd3732-ovnkube-script-lib\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.314338 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.311039 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8fz2\" (UniqueName: \"kubernetes.io/projected/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-kube-api-access-f8fz2\") pod \"network-metrics-daemon-tt8gq\" (UID: \"a1f5a77d-43d5-48bc-a8b5-5db8f1152853\") " pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:44:40.314338 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.311062 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-cnibin\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.314338 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.311081 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5705a40-d3e4-40a8-a561-17c435f388b1-system-cni-dir\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:44:40.314338 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.311086 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-host-var-lib-kubelet\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.314338 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.311098 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-etc-modprobe-d\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.314338 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.311117 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6cd67a78-09f1-4988-99b0-1ac9097f2f39-etc-selinux\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:44:40.314338 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.311125 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-host-var-lib-kubelet\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.314338 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.311143 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-lib-modules\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.314338 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.311146 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-node-log\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.314338 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.311163 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-kubelet\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.314840 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.311168 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5705a40-d3e4-40a8-a561-17c435f388b1-cnibin\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:44:40.314840 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.311211 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-cnibin\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.314840 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.311319 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7dc7486d-3188-4bb8-b907-b653caa18660-kubelet-config\") pod \"global-pull-secret-syncer-r5l82\" (UID: \"7dc7486d-3188-4bb8-b907-b653caa18660\") " pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:44:40.314840 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.311377 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6cd67a78-09f1-4988-99b0-1ac9097f2f39-etc-selinux\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:44:40.314840 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.311375 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-lib-modules\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.314840 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.311502 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5705a40-d3e4-40a8-a561-17c435f388b1-cnibin\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:44:40.314840 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.311722 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/197fb18c-6703-4deb-93b3-0186d7ceed67-konnectivity-ca\") pod \"konnectivity-agent-n7mks\" (UID: \"197fb18c-6703-4deb-93b3-0186d7ceed67\") " pod="kube-system/konnectivity-agent-n7mks" Nov 01 05:44:40.314840 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.312040 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/21519345-86b8-441e-a395-1e8385cd3732-env-overrides\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.314840 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.312078 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/67ae987f-79b9-4852-98bd-e0537c412b17-multus-daemon-config\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.314840 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.312236 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cde21439-a494-47cd-8cc5-161fc96a6701-serviceca\") pod \"node-ca-tw96g\" (UID: \"cde21439-a494-47cd-8cc5-161fc96a6701\") " pod="openshift-image-registry/node-ca-tw96g" Nov 01 05:44:40.314840 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.312248 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/21519345-86b8-441e-a395-1e8385cd3732-ovnkube-script-lib\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.314840 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.314026 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2acbc647-2361-4935-b503-5bd9af74f822-etc-tuned\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.314840 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.314207 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2acbc647-2361-4935-b503-5bd9af74f822-tmp\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.314840 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.314242 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/21519345-86b8-441e-a395-1e8385cd3732-ovn-node-metrics-cert\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.314840 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.314317 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/197fb18c-6703-4deb-93b3-0186d7ceed67-agent-certs\") pod \"konnectivity-agent-n7mks\" (UID: \"197fb18c-6703-4deb-93b3-0186d7ceed67\") " pod="kube-system/konnectivity-agent-n7mks" Nov 01 05:44:40.368790 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:40.368767 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 05:44:40.368790 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:40.368793 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 05:44:40.368999 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:40.368807 2586 projected.go:196] Error preparing data for projected volume kube-api-access-k2xnh for pod openshift-network-diagnostics/network-check-target-ljcjx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 05:44:40.368999 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:40.368865 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/529e2d97-b167-4011-a215-02ece8519c00-kube-api-access-k2xnh podName:529e2d97-b167-4011-a215-02ece8519c00 nodeName:}" failed. No retries permitted until 2026-11-01 05:44:40.868847322 +0000 UTC m=+3.170472931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-k2xnh" (UniqueName: "kubernetes.io/projected/529e2d97-b167-4011-a215-02ece8519c00-kube-api-access-k2xnh") pod "network-check-target-ljcjx" (UID: "529e2d97-b167-4011-a215-02ece8519c00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 05:44:40.371173 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.371146 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdrd9\" (UniqueName: \"kubernetes.io/projected/0719dc90-dbae-425a-8439-812425e1b54c-kube-api-access-mdrd9\") pod \"iptables-alerter-qbg2s\" (UID: \"0719dc90-dbae-425a-8439-812425e1b54c\") " pod="openshift-network-operator/iptables-alerter-qbg2s" Nov 01 05:44:40.371840 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.371614 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrtw2\" (UniqueName: \"kubernetes.io/projected/ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8-kube-api-access-rrtw2\") pod \"node-resolver-2qtwx\" (UID: \"ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8\") " pod="openshift-dns/node-resolver-2qtwx" Nov 01 05:44:40.371840 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.371787 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6xv7\" (UniqueName: \"kubernetes.io/projected/21519345-86b8-441e-a395-1e8385cd3732-kube-api-access-g6xv7\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.371840 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.371818 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8fz2\" (UniqueName: \"kubernetes.io/projected/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-kube-api-access-f8fz2\") pod \"network-metrics-daemon-tt8gq\" (UID: \"a1f5a77d-43d5-48bc-a8b5-5db8f1152853\") " pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:44:40.371976 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.371940 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s56k\" (UniqueName: \"kubernetes.io/projected/2acbc647-2361-4935-b503-5bd9af74f822-kube-api-access-4s56k\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.372315 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.372293 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz6tw\" (UniqueName: \"kubernetes.io/projected/6cd67a78-09f1-4988-99b0-1ac9097f2f39-kube-api-access-kz6tw\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:44:40.372399 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.372381 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gbgq\" (UniqueName: \"kubernetes.io/projected/67ae987f-79b9-4852-98bd-e0537c412b17-kube-api-access-7gbgq\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.372610 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.372596 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dbh2\" (UniqueName: \"kubernetes.io/projected/cde21439-a494-47cd-8cc5-161fc96a6701-kube-api-access-8dbh2\") pod \"node-ca-tw96g\" (UID: \"cde21439-a494-47cd-8cc5-161fc96a6701\") " pod="openshift-image-registry/node-ca-tw96g" Nov 01 05:44:40.372976 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.372960 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzxml\" (UniqueName: \"kubernetes.io/projected/f5705a40-d3e4-40a8-a561-17c435f388b1-kube-api-access-dzxml\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:44:40.480216 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.480184 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:44:40.490667 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:40.490642 2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2acbc647_2361_4935_b503_5bd9af74f822.slice/crio-cfb904abac59b4c1cd142c40f5790f5c159840b3a6291d8cbd73fb666e47552a WatchSource:0}: Error finding container cfb904abac59b4c1cd142c40f5790f5c159840b3a6291d8cbd73fb666e47552a: Status 404 returned error can't find the container with id cfb904abac59b4c1cd142c40f5790f5c159840b3a6291d8cbd73fb666e47552a Nov 01 05:44:40.497146 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.497132 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tw96g" Nov 01 05:44:40.504774 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:40.504752 2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde21439_a494_47cd_8cc5_161fc96a6701.slice/crio-a3c54009ff4e69efff07e45faa1e917b0aec9d2b2f3fd61935987368b12409c5 WatchSource:0}: Error finding container a3c54009ff4e69efff07e45faa1e917b0aec9d2b2f3fd61935987368b12409c5: Status 404 returned error can't find the container with id a3c54009ff4e69efff07e45faa1e917b0aec9d2b2f3fd61935987368b12409c5 Nov 01 05:44:40.509405 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.509367 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:44:40.509499 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.509441 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qh47z" Nov 01 05:44:40.513117 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.513103 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n7mks" Nov 01 05:44:40.519618 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:40.519591 2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21519345_86b8_441e_a395_1e8385cd3732.slice/crio-0f3e2e028ef2253cf0b2d40ba7a04a569991030505557c23570fea3e72cd9bb6 WatchSource:0}: Error finding container 0f3e2e028ef2253cf0b2d40ba7a04a569991030505557c23570fea3e72cd9bb6: Status 404 returned error can't find the container with id 0f3e2e028ef2253cf0b2d40ba7a04a569991030505557c23570fea3e72cd9bb6 Nov 01 05:44:40.521983 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:40.521960 2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ae987f_79b9_4852_98bd_e0537c412b17.slice/crio-3b51306f82e344f59688bfb3a69b3ac0b052bad3fd4178b3b55cbba30d25e728 WatchSource:0}: Error finding container 3b51306f82e344f59688bfb3a69b3ac0b052bad3fd4178b3b55cbba30d25e728: Status 404 returned error can't find the container with id 3b51306f82e344f59688bfb3a69b3ac0b052bad3fd4178b3b55cbba30d25e728 Nov 01 05:44:40.523129 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.523070 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:44:40.524798 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:40.524779 2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod197fb18c_6703_4deb_93b3_0186d7ceed67.slice/crio-bf8defe53821b5fe41a5e5da1523fee6fa713a3253535d5f90f74b6afe37c04c WatchSource:0}: Error finding container bf8defe53821b5fe41a5e5da1523fee6fa713a3253535d5f90f74b6afe37c04c: Status 404 returned error can't find the container with id bf8defe53821b5fe41a5e5da1523fee6fa713a3253535d5f90f74b6afe37c04c Nov 01 05:44:40.525307 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.525288 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qbg2s" Nov 01 05:44:40.529944 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.529929 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2qtwx" Nov 01 05:44:40.532070 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.531972 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:44:40.532257 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:40.532238 2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd67a78_09f1_4988_99b0_1ac9097f2f39.slice/crio-40592ff5d356deee21ad5769e9533ff695bdf1eb214a9155399cf35ce47382eb WatchSource:0}: Error finding container 40592ff5d356deee21ad5769e9533ff695bdf1eb214a9155399cf35ce47382eb: Status 404 returned error can't find the container with id 40592ff5d356deee21ad5769e9533ff695bdf1eb214a9155399cf35ce47382eb Nov 01 05:44:40.534216 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:40.534197 2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0719dc90_dbae_425a_8439_812425e1b54c.slice/crio-6d7d36ac248e6bfb113ab4cda9fbc9207bd58b97e154ecad53c594f39f56d265 WatchSource:0}: Error finding container 6d7d36ac248e6bfb113ab4cda9fbc9207bd58b97e154ecad53c594f39f56d265: Status 404 returned error can't find the container with id 6d7d36ac248e6bfb113ab4cda9fbc9207bd58b97e154ecad53c594f39f56d265 Nov 01 05:44:40.540687 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:40.540668 2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec7c5307_5d9b_4b92_b4e3_35d964ba4cd8.slice/crio-944070a7add267d0399adb6828fa1d1240081b78d60309c0db13f195a30023b9 WatchSource:0}: Error finding container 944070a7add267d0399adb6828fa1d1240081b78d60309c0db13f195a30023b9: Status 404 returned error can't find the container with id 944070a7add267d0399adb6828fa1d1240081b78d60309c0db13f195a30023b9 Nov 01 05:44:40.541247 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:44:40.541230 2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5705a40_d3e4_40a8_a561_17c435f388b1.slice/crio-746049f9b24a68fedc050b50f4332cafda0a4d614d8feaa7cc941e321dcf62c9 WatchSource:0}: Error finding container 746049f9b24a68fedc050b50f4332cafda0a4d614d8feaa7cc941e321dcf62c9: Status 404 returned error can't find the container with id 746049f9b24a68fedc050b50f4332cafda0a4d614d8feaa7cc941e321dcf62c9 Nov 01 05:44:40.816375 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.816346 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7dc7486d-3188-4bb8-b907-b653caa18660-original-pull-secret\") pod \"global-pull-secret-syncer-r5l82\" (UID: \"7dc7486d-3188-4bb8-b907-b653caa18660\") " pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:44:40.816561 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.816383 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs\") pod \"network-metrics-daemon-tt8gq\" (UID: \"a1f5a77d-43d5-48bc-a8b5-5db8f1152853\") " pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:44:40.816561 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:40.816507 2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 05:44:40.816561 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:40.816545 2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 05:44:40.816717 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:40.816555 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dc7486d-3188-4bb8-b907-b653caa18660-original-pull-secret podName:7dc7486d-3188-4bb8-b907-b653caa18660 nodeName:}" failed. No retries permitted until 2026-11-01 05:44:41.816542018 +0000 UTC m=+4.118167613 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7dc7486d-3188-4bb8-b907-b653caa18660-original-pull-secret") pod "global-pull-secret-syncer-r5l82" (UID: "7dc7486d-3188-4bb8-b907-b653caa18660") : object "kube-system"/"original-pull-secret" not registered Nov 01 05:44:40.816717 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:40.816625 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs podName:a1f5a77d-43d5-48bc-a8b5-5db8f1152853 nodeName:}" failed. No retries permitted until 2026-11-01 05:44:41.816606764 +0000 UTC m=+4.118232366 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs") pod "network-metrics-daemon-tt8gq" (UID: "a1f5a77d-43d5-48bc-a8b5-5db8f1152853") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 05:44:40.917010 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:40.916980 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2xnh\" (UniqueName: \"kubernetes.io/projected/529e2d97-b167-4011-a215-02ece8519c00-kube-api-access-k2xnh\") pod \"network-check-target-ljcjx\" (UID: \"529e2d97-b167-4011-a215-02ece8519c00\") " pod="openshift-network-diagnostics/network-check-target-ljcjx" Nov 01 05:44:40.917194 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:40.917164 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 05:44:40.917194 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:40.917191 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 05:44:40.917305 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:40.917207 2586 projected.go:196] Error preparing data for projected volume kube-api-access-k2xnh for pod openshift-network-diagnostics/network-check-target-ljcjx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 05:44:40.917305 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:40.917268 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/529e2d97-b167-4011-a215-02ece8519c00-kube-api-access-k2xnh podName:529e2d97-b167-4011-a215-02ece8519c00 nodeName:}" failed. No retries permitted until 2026-11-01 05:44:41.917249803 +0000 UTC m=+4.218875408 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-k2xnh" (UniqueName: "kubernetes.io/projected/529e2d97-b167-4011-a215-02ece8519c00-kube-api-access-k2xnh") pod "network-check-target-ljcjx" (UID: "529e2d97-b167-4011-a215-02ece8519c00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 05:44:41.180564 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:41.180462 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" event={"ID":"6cd67a78-09f1-4988-99b0-1ac9097f2f39","Type":"ContainerStarted","Data":"40592ff5d356deee21ad5769e9533ff695bdf1eb214a9155399cf35ce47382eb"} Nov 01 05:44:41.181847 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:41.181792 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n7mks" event={"ID":"197fb18c-6703-4deb-93b3-0186d7ceed67","Type":"ContainerStarted","Data":"bf8defe53821b5fe41a5e5da1523fee6fa713a3253535d5f90f74b6afe37c04c"} Nov 01 05:44:41.183655 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:41.183626 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cb599" event={"ID":"2acbc647-2361-4935-b503-5bd9af74f822","Type":"ContainerStarted","Data":"cfb904abac59b4c1cd142c40f5790f5c159840b3a6291d8cbd73fb666e47552a"} Nov 01 05:44:41.185048 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:41.185024 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7rrq" event={"ID":"f5705a40-d3e4-40a8-a561-17c435f388b1","Type":"ContainerStarted","Data":"746049f9b24a68fedc050b50f4332cafda0a4d614d8feaa7cc941e321dcf62c9"} Nov 01 05:44:41.186273 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:41.186246 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2qtwx" event={"ID":"ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8","Type":"ContainerStarted","Data":"944070a7add267d0399adb6828fa1d1240081b78d60309c0db13f195a30023b9"} Nov 01 05:44:41.187659 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:41.187638 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qbg2s" event={"ID":"0719dc90-dbae-425a-8439-812425e1b54c","Type":"ContainerStarted","Data":"6d7d36ac248e6bfb113ab4cda9fbc9207bd58b97e154ecad53c594f39f56d265"} Nov 01 05:44:41.189958 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:41.189933 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qh47z" event={"ID":"67ae987f-79b9-4852-98bd-e0537c412b17","Type":"ContainerStarted","Data":"3b51306f82e344f59688bfb3a69b3ac0b052bad3fd4178b3b55cbba30d25e728"} Nov 01 05:44:41.191676 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:41.191653 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" event={"ID":"21519345-86b8-441e-a395-1e8385cd3732","Type":"ContainerStarted","Data":"0f3e2e028ef2253cf0b2d40ba7a04a569991030505557c23570fea3e72cd9bb6"} Nov 01 05:44:41.192824 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:41.192804 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tw96g" event={"ID":"cde21439-a494-47cd-8cc5-161fc96a6701","Type":"ContainerStarted","Data":"a3c54009ff4e69efff07e45faa1e917b0aec9d2b2f3fd61935987368b12409c5"} Nov 01 05:44:41.273241 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:41.273197 2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 05:39:40 +0000 UTC" deadline="2028-05-28 09:24:58.658988828 +0000 UTC" Nov 01 05:44:41.273241 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:41.273234 2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13779h40m17.385759727s" Nov 01 05:44:41.824124 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:41.824088 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7dc7486d-3188-4bb8-b907-b653caa18660-original-pull-secret\") pod \"global-pull-secret-syncer-r5l82\" (UID: \"7dc7486d-3188-4bb8-b907-b653caa18660\") " pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:44:41.824316 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:41.824148 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs\") pod \"network-metrics-daemon-tt8gq\" (UID: \"a1f5a77d-43d5-48bc-a8b5-5db8f1152853\") " pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:44:41.824316 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:41.824245 2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 05:44:41.824316 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:41.824311 2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 05:44:41.824496 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:41.824321 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dc7486d-3188-4bb8-b907-b653caa18660-original-pull-secret podName:7dc7486d-3188-4bb8-b907-b653caa18660 nodeName:}" failed. No retries permitted until 2026-11-01 05:44:43.82430192 +0000 UTC m=+6.125927515 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7dc7486d-3188-4bb8-b907-b653caa18660-original-pull-secret") pod "global-pull-secret-syncer-r5l82" (UID: "7dc7486d-3188-4bb8-b907-b653caa18660") : object "kube-system"/"original-pull-secret" not registered Nov 01 05:44:41.824496 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:41.824362 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs podName:a1f5a77d-43d5-48bc-a8b5-5db8f1152853 nodeName:}" failed. No retries permitted until 2026-11-01 05:44:43.824347183 +0000 UTC m=+6.125972780 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs") pod "network-metrics-daemon-tt8gq" (UID: "a1f5a77d-43d5-48bc-a8b5-5db8f1152853") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 05:44:41.925555 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:41.925518 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2xnh\" (UniqueName: \"kubernetes.io/projected/529e2d97-b167-4011-a215-02ece8519c00-kube-api-access-k2xnh\") pod \"network-check-target-ljcjx\" (UID: \"529e2d97-b167-4011-a215-02ece8519c00\") " pod="openshift-network-diagnostics/network-check-target-ljcjx" Nov 01 05:44:41.926043 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:41.925775 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 05:44:41.926043 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:41.925797 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 05:44:41.926043 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:41.925810 2586 projected.go:196] Error preparing data for projected volume kube-api-access-k2xnh for pod openshift-network-diagnostics/network-check-target-ljcjx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 05:44:41.926043 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:41.925866 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/529e2d97-b167-4011-a215-02ece8519c00-kube-api-access-k2xnh podName:529e2d97-b167-4011-a215-02ece8519c00 nodeName:}" failed. No retries permitted until 2026-11-01 05:44:43.925846746 +0000 UTC m=+6.227472340 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-k2xnh" (UniqueName: "kubernetes.io/projected/529e2d97-b167-4011-a215-02ece8519c00-kube-api-access-k2xnh") pod "network-check-target-ljcjx" (UID: "529e2d97-b167-4011-a215-02ece8519c00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 05:44:42.172669 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:42.172600 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:44:42.172811 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:42.172749 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tt8gq" podUID="a1f5a77d-43d5-48bc-a8b5-5db8f1152853" Nov 01 05:44:42.172811 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:42.172613 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:44:42.172924 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:42.172903 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5l82" podUID="7dc7486d-3188-4bb8-b907-b653caa18660" Nov 01 05:44:42.172978 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:42.172940 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljcjx" Nov 01 05:44:42.173029 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:42.173007 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljcjx" podUID="529e2d97-b167-4011-a215-02ece8519c00" Nov 01 05:44:42.274105 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:42.274004 2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 05:39:40 +0000 UTC" deadline="2028-06-27 04:16:22.142992772 +0000 UTC" Nov 01 05:44:42.274105 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:42.274064 2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14494h31m39.868932561s" Nov 01 05:44:43.844962 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:43.844237 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs\") pod \"network-metrics-daemon-tt8gq\" (UID: \"a1f5a77d-43d5-48bc-a8b5-5db8f1152853\") " pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:44:43.844962 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:43.844539 2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 05:44:43.844962 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:43.844577 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7dc7486d-3188-4bb8-b907-b653caa18660-original-pull-secret\") pod \"global-pull-secret-syncer-r5l82\" (UID: \"7dc7486d-3188-4bb8-b907-b653caa18660\") " pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:44:43.844962 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:43.844611 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs podName:a1f5a77d-43d5-48bc-a8b5-5db8f1152853 nodeName:}" failed. No retries permitted until 2026-11-01 05:44:47.84459195 +0000 UTC m=+10.146217554 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs") pod "network-metrics-daemon-tt8gq" (UID: "a1f5a77d-43d5-48bc-a8b5-5db8f1152853") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 05:44:43.844962 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:43.844689 2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 05:44:43.844962 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:43.844738 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dc7486d-3188-4bb8-b907-b653caa18660-original-pull-secret podName:7dc7486d-3188-4bb8-b907-b653caa18660 nodeName:}" failed. No retries permitted until 2026-11-01 05:44:47.844723971 +0000 UTC m=+10.146349566 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7dc7486d-3188-4bb8-b907-b653caa18660-original-pull-secret") pod "global-pull-secret-syncer-r5l82" (UID: "7dc7486d-3188-4bb8-b907-b653caa18660") : object "kube-system"/"original-pull-secret" not registered Nov 01 05:44:43.945103 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:43.945026 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2xnh\" (UniqueName: \"kubernetes.io/projected/529e2d97-b167-4011-a215-02ece8519c00-kube-api-access-k2xnh\") pod \"network-check-target-ljcjx\" (UID: \"529e2d97-b167-4011-a215-02ece8519c00\") " pod="openshift-network-diagnostics/network-check-target-ljcjx" Nov 01 05:44:43.945277 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:43.945226 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 05:44:43.945277 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:43.945247 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 05:44:43.945277 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:43.945260 2586 projected.go:196] Error preparing data for projected volume kube-api-access-k2xnh for pod openshift-network-diagnostics/network-check-target-ljcjx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 05:44:43.945432 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:43.945318 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/529e2d97-b167-4011-a215-02ece8519c00-kube-api-access-k2xnh podName:529e2d97-b167-4011-a215-02ece8519c00 nodeName:}" failed. No retries permitted until 2026-11-01 05:44:47.945300856 +0000 UTC m=+10.246926451 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-k2xnh" (UniqueName: "kubernetes.io/projected/529e2d97-b167-4011-a215-02ece8519c00-kube-api-access-k2xnh") pod "network-check-target-ljcjx" (UID: "529e2d97-b167-4011-a215-02ece8519c00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 05:44:44.172346 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:44.172283 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljcjx" Nov 01 05:44:44.172516 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:44.172418 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljcjx" podUID="529e2d97-b167-4011-a215-02ece8519c00" Nov 01 05:44:44.172836 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:44.172799 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:44:44.172953 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:44.172903 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tt8gq" podUID="a1f5a77d-43d5-48bc-a8b5-5db8f1152853" Nov 01 05:44:44.172953 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:44.172934 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:44:44.173055 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:44.173005 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5l82" podUID="7dc7486d-3188-4bb8-b907-b653caa18660" Nov 01 05:44:44.236138 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:44.236033 2586 generic.go:356] "Generic (PLEG): container finished" podID="ac965f1cfbc1129016d728e686326ed6" containerID="cdd5aa48e73883f04b05c4608010e07f9acc13cc79baa6b38526ce5f538f07b4" exitCode=0 Nov 01 05:44:44.236138 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:44.236116 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal" event={"ID":"ac965f1cfbc1129016d728e686326ed6","Type":"ContainerDied","Data":"cdd5aa48e73883f04b05c4608010e07f9acc13cc79baa6b38526ce5f538f07b4"} Nov 01 05:44:44.242178 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:44.241641 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-195.us-west-1.compute.internal" event={"ID":"c656c4baa8463ac8d24886e04ca99821","Type":"ContainerStarted","Data":"f4d5b687ca74a3185a37acbcf44ddff111db767b33ad49551a003d7af8c260aa"} Nov 01 05:44:45.248904 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:45.248858 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal" event={"ID":"ac965f1cfbc1129016d728e686326ed6","Type":"ContainerStarted","Data":"edfd3bedf5e2f01e7b42ca78b9b03f8b95a3df681b1189d6d058e8e2aca61e21"} Nov 01 05:44:45.254825 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:45.254799 2586 generic.go:356] "Generic (PLEG): container finished" podID="f5705a40-d3e4-40a8-a561-17c435f388b1" containerID="f77c4d4e94b3626e04c5958abfcf2031137d599ca921225f62e442f996addec4" exitCode=0 Nov 01 05:44:45.255883 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:45.255858 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7rrq" event={"ID":"f5705a40-d3e4-40a8-a561-17c435f388b1","Type":"ContainerDied","Data":"f77c4d4e94b3626e04c5958abfcf2031137d599ca921225f62e442f996addec4"} Nov 01 05:44:45.365375 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:45.364766 2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-195.us-west-1.compute.internal" podStartSLOduration=6.364746443 podStartE2EDuration="6.364746443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:44:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:44:44.468371879 +0000 UTC m=+6.769997492" watchObservedRunningTime="2026-11-01 05:44:45.364746443 +0000 UTC m=+7.666372056" Nov 01 05:44:45.486241 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:45.485894 2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal" podStartSLOduration=6.48588013 podStartE2EDuration="6.48588013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:44:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:44:45.364979049 +0000 UTC m=+7.666604664" watchObservedRunningTime="2026-11-01 05:44:45.48588013 +0000 UTC m=+7.787505742" Nov 01 05:44:46.173634 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:46.173609 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:44:46.173760 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:46.173739 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tt8gq" podUID="a1f5a77d-43d5-48bc-a8b5-5db8f1152853" Nov 01 05:44:46.173832 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:46.173811 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:44:46.173891 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:46.173875 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5l82" podUID="7dc7486d-3188-4bb8-b907-b653caa18660" Nov 01 05:44:46.175078 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:46.175056 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljcjx" Nov 01 05:44:46.175181 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:46.175153 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljcjx" podUID="529e2d97-b167-4011-a215-02ece8519c00" Nov 01 05:44:46.265339 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:46.265305 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n7mks" event={"ID":"197fb18c-6703-4deb-93b3-0186d7ceed67","Type":"ContainerStarted","Data":"2174f299eaaa1c7d554b316b62d8f9e44a13d10a8c51e85c00edab46e653da86"} Nov 01 05:44:47.268064 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:47.268027 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tw96g" event={"ID":"cde21439-a494-47cd-8cc5-161fc96a6701","Type":"ContainerStarted","Data":"b7bd7a02f96361af8533b1070960ef68c75f0b3e96510f46b4f903b54e3b2935"} Nov 01 05:44:47.272141 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:47.272118 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" event={"ID":"6cd67a78-09f1-4988-99b0-1ac9097f2f39","Type":"ContainerStarted","Data":"22b49080c2d49aa415442c6b6e6f485fdadea0503e2d9af2c9f4a561e95e2991"} Nov 01 05:44:47.384042 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:47.383941 2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tw96g" podStartSLOduration=3.73542822 podStartE2EDuration="9.383927631s" totalImagesPullingTime="5.648499411s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:44:47.383712731 +0000 UTC m=+9.685338343" watchObservedRunningTime="2026-11-01 05:44:47.383927631 +0000 UTC m=+9.685553293" Nov 01 05:44:47.384243 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:47.384216 2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-n7mks" podStartSLOduration=4.820457162 podStartE2EDuration="9.384208124s" totalImagesPullingTime="4.563750962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:44:46.380741669 +0000 UTC m=+8.682367283" watchObservedRunningTime="2026-11-01 05:44:47.384208124 +0000 UTC m=+9.685833735" Nov 01 05:44:47.881236 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:47.881190 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs\") pod \"network-metrics-daemon-tt8gq\" (UID: \"a1f5a77d-43d5-48bc-a8b5-5db8f1152853\") " pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:44:47.881422 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:47.881327 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7dc7486d-3188-4bb8-b907-b653caa18660-original-pull-secret\") pod \"global-pull-secret-syncer-r5l82\" (UID: \"7dc7486d-3188-4bb8-b907-b653caa18660\") " pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:44:47.882181 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:47.881552 2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 05:44:47.882181 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:47.881622 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dc7486d-3188-4bb8-b907-b653caa18660-original-pull-secret podName:7dc7486d-3188-4bb8-b907-b653caa18660 nodeName:}" failed. No retries permitted until 2026-11-01 05:44:55.881603785 +0000 UTC m=+18.183229393 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7dc7486d-3188-4bb8-b907-b653caa18660-original-pull-secret") pod "global-pull-secret-syncer-r5l82" (UID: "7dc7486d-3188-4bb8-b907-b653caa18660") : object "kube-system"/"original-pull-secret" not registered Nov 01 05:44:47.882181 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:47.882031 2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 05:44:47.882181 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:47.882099 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs podName:a1f5a77d-43d5-48bc-a8b5-5db8f1152853 nodeName:}" failed. No retries permitted until 2026-11-01 05:44:55.882083578 +0000 UTC m=+18.183709224 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs") pod "network-metrics-daemon-tt8gq" (UID: "a1f5a77d-43d5-48bc-a8b5-5db8f1152853") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 05:44:47.982862 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:47.982541 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2xnh\" (UniqueName: \"kubernetes.io/projected/529e2d97-b167-4011-a215-02ece8519c00-kube-api-access-k2xnh\") pod \"network-check-target-ljcjx\" (UID: \"529e2d97-b167-4011-a215-02ece8519c00\") " pod="openshift-network-diagnostics/network-check-target-ljcjx" Nov 01 05:44:47.982862 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:47.982710 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 05:44:47.982862 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:47.982754 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 05:44:47.982862 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:47.982767 2586 projected.go:196] Error preparing data for projected volume kube-api-access-k2xnh for pod openshift-network-diagnostics/network-check-target-ljcjx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 05:44:47.982862 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:47.982823 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/529e2d97-b167-4011-a215-02ece8519c00-kube-api-access-k2xnh podName:529e2d97-b167-4011-a215-02ece8519c00 nodeName:}" failed. No retries permitted until 2026-11-01 05:44:55.982805835 +0000 UTC m=+18.284431442 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-k2xnh" (UniqueName: "kubernetes.io/projected/529e2d97-b167-4011-a215-02ece8519c00-kube-api-access-k2xnh") pod "network-check-target-ljcjx" (UID: "529e2d97-b167-4011-a215-02ece8519c00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 05:44:48.172725 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:48.172651 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:44:48.172868 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:48.172740 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tt8gq" podUID="a1f5a77d-43d5-48bc-a8b5-5db8f1152853" Nov 01 05:44:48.173259 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:48.172994 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:44:48.173259 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:48.173113 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljcjx" Nov 01 05:44:48.173259 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:48.173133 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5l82" podUID="7dc7486d-3188-4bb8-b907-b653caa18660" Nov 01 05:44:48.173259 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:48.173196 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljcjx" podUID="529e2d97-b167-4011-a215-02ece8519c00" Nov 01 05:44:48.783159 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:48.783118 2586 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 05:44:49.168413 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:49.168253 2586 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T05:44:48.783140228Z","UUID":"f41ca4d9-1c5c-4d07-9445-d85e288a933b","Handler":null,"Name":"","Endpoint":""} Nov 01 05:44:49.169959 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:49.169926 2586 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 05:44:49.170081 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:49.169984 2586 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 05:44:49.177353 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:49.177141 2586 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-n7mks" Nov 01 05:44:49.177596 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:49.177573 2586 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-n7mks" Nov 01 05:44:49.279323 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:49.279285 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2qtwx" event={"ID":"ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8","Type":"ContainerStarted","Data":"979d0dc8e655efb13e1d258daaac5655370de3fbc63da85ed7f9cc9e9c287edc"} Nov 01 05:44:49.281916 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:49.281883 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qh47z" event={"ID":"67ae987f-79b9-4852-98bd-e0537c412b17","Type":"ContainerStarted","Data":"24dc5b6d79fe722fd39c3d892f49e4fde01f5b83bf9bf74ee8ccd22e3cfd9168"} Nov 01 05:44:49.284169 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:49.284141 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" event={"ID":"6cd67a78-09f1-4988-99b0-1ac9097f2f39","Type":"ContainerStarted","Data":"975f4a9ed1570a156693b47bb1a17e18fcd283fed8d1f24fb795316fe9c0fa32"} Nov 01 05:44:49.409784 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:49.409729 2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-2qtwx" podStartSLOduration=2.088110797 podStartE2EDuration="10.409715055s" totalImagesPullingTime="8.321604258s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:44:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:44:49.409653328 +0000 UTC m=+11.711278943" watchObservedRunningTime="2026-11-01 05:44:49.409715055 +0000 UTC m=+11.711340668" Nov 01 05:44:49.527361 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:49.527304 2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-qh47z" podStartSLOduration=3.368505803 podStartE2EDuration="11.527286553s" totalImagesPullingTime="8.15878075s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:44:49.526665578 +0000 UTC m=+11.828291194" watchObservedRunningTime="2026-11-01 05:44:49.527286553 +0000 UTC m=+11.828912166" Nov 01 05:44:50.172121 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:50.172093 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:44:50.172530 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:50.172149 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljcjx" Nov 01 05:44:50.172530 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:50.172229 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:44:50.172530 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:50.172252 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tt8gq" podUID="a1f5a77d-43d5-48bc-a8b5-5db8f1152853" Nov 01 05:44:50.172530 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:50.172309 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5l82" podUID="7dc7486d-3188-4bb8-b907-b653caa18660" Nov 01 05:44:50.172530 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:50.172375 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljcjx" podUID="529e2d97-b167-4011-a215-02ece8519c00" Nov 01 05:44:50.292040 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:50.292001 2586 generic.go:356] "Generic (PLEG): container finished" podID="f5705a40-d3e4-40a8-a561-17c435f388b1" containerID="b23ee54876e3a909f039a31bf3bee7bec29cf7bfcbd5e962789257ddfc904063" exitCode=0 Nov 01 05:44:50.292174 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:50.292089 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7rrq" event={"ID":"f5705a40-d3e4-40a8-a561-17c435f388b1","Type":"ContainerDied","Data":"b23ee54876e3a909f039a31bf3bee7bec29cf7bfcbd5e962789257ddfc904063"} Nov 01 05:44:50.294995 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:50.294974 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qbg2s" event={"ID":"0719dc90-dbae-425a-8439-812425e1b54c","Type":"ContainerStarted","Data":"d26eb79728bc7d472a7188e80056a0bda745c37b77b3f97e53c8ed374ff6314e"} Nov 01 05:44:50.298289 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:50.298261 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" event={"ID":"6cd67a78-09f1-4988-99b0-1ac9097f2f39","Type":"ContainerStarted","Data":"2a649260e461045b77b0c9c7cdf74be14d583c0ecb7069e357aee00d4d584c4c"} Nov 01 05:44:50.562099 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:50.562000 2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" podStartSLOduration=3.249046935 podStartE2EDuration="12.561982483s" totalImagesPullingTime="9.312935548s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:44:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:44:50.561561246 +0000 UTC m=+12.863186858" watchObservedRunningTime="2026-11-01 05:44:50.561982483 +0000 UTC m=+12.863608092" Nov 01 05:44:50.674289 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:50.674250 2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qbg2s" podStartSLOduration=4.649021223 podStartE2EDuration="12.674237652s" totalImagesPullingTime="8.025216429s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:44:50.674019894 +0000 UTC m=+12.975645506" watchObservedRunningTime="2026-11-01 05:44:50.674237652 +0000 UTC m=+12.975863307" Nov 01 05:44:51.302216 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:51.302147 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cb599" event={"ID":"2acbc647-2361-4935-b503-5bd9af74f822","Type":"ContainerStarted","Data":"7985cb5a17b0f4f37194349da1246da46de75d0fcdb729da033c49d3fdc41485"} Nov 01 05:44:51.419094 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:51.419046 2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-cb599" podStartSLOduration=3.292785849 podStartE2EDuration="13.419030901s" totalImagesPullingTime="10.126245052s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:44:51.418318215 +0000 UTC m=+13.719943821" watchObservedRunningTime="2026-11-01 05:44:51.419030901 +0000 UTC m=+13.720656533" Nov 01 05:44:52.172441 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:52.172403 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:44:52.172613 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:52.172467 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:44:52.172613 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:52.172471 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljcjx" Nov 01 05:44:52.172613 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:52.172584 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tt8gq" podUID="a1f5a77d-43d5-48bc-a8b5-5db8f1152853" Nov 01 05:44:52.172723 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:52.172697 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5l82" podUID="7dc7486d-3188-4bb8-b907-b653caa18660" Nov 01 05:44:52.172772 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:52.172757 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljcjx" podUID="529e2d97-b167-4011-a215-02ece8519c00" Nov 01 05:44:52.306921 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:52.306780 2586 generic.go:356] "Generic (PLEG): container finished" podID="f5705a40-d3e4-40a8-a561-17c435f388b1" containerID="cb0e98cc1cdcfd74db81edd1ce2d1a4b01e6177883ac74f7a85b730ef8e3d160" exitCode=0 Nov 01 05:44:52.306921 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:52.306882 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7rrq" event={"ID":"f5705a40-d3e4-40a8-a561-17c435f388b1","Type":"ContainerDied","Data":"cb0e98cc1cdcfd74db81edd1ce2d1a4b01e6177883ac74f7a85b730ef8e3d160"} Nov 01 05:44:54.172716 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:54.172399 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:44:54.172716 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:54.172497 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljcjx" Nov 01 05:44:54.173254 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:54.172728 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tt8gq" podUID="a1f5a77d-43d5-48bc-a8b5-5db8f1152853" Nov 01 05:44:54.173254 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:54.172548 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:44:54.173254 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:54.172800 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljcjx" podUID="529e2d97-b167-4011-a215-02ece8519c00" Nov 01 05:44:54.173254 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:54.172900 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5l82" podUID="7dc7486d-3188-4bb8-b907-b653caa18660" Nov 01 05:44:54.314442 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:54.314407 2586 generic.go:356] "Generic (PLEG): container finished" podID="f5705a40-d3e4-40a8-a561-17c435f388b1" containerID="1801d85bfe506b0c3f992052aa347d3f4c3cb27f80a607e858c0c125402a8ae6" exitCode=0 Nov 01 05:44:54.314611 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:54.314473 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7rrq" event={"ID":"f5705a40-d3e4-40a8-a561-17c435f388b1","Type":"ContainerDied","Data":"1801d85bfe506b0c3f992052aa347d3f4c3cb27f80a607e858c0c125402a8ae6"} Nov 01 05:44:55.944837 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:55.944803 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs\") pod \"network-metrics-daemon-tt8gq\" (UID: \"a1f5a77d-43d5-48bc-a8b5-5db8f1152853\") " pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:44:55.945250 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:55.944866 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7dc7486d-3188-4bb8-b907-b653caa18660-original-pull-secret\") pod \"global-pull-secret-syncer-r5l82\" (UID: \"7dc7486d-3188-4bb8-b907-b653caa18660\") " pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:44:55.945250 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:55.944953 2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 05:44:55.945250 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:55.944956 2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 05:44:55.945250 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:55.945003 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dc7486d-3188-4bb8-b907-b653caa18660-original-pull-secret podName:7dc7486d-3188-4bb8-b907-b653caa18660 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:11.944991677 +0000 UTC m=+34.246617267 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7dc7486d-3188-4bb8-b907-b653caa18660-original-pull-secret") pod "global-pull-secret-syncer-r5l82" (UID: "7dc7486d-3188-4bb8-b907-b653caa18660") : object "kube-system"/"original-pull-secret" not registered Nov 01 05:44:55.945250 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:55.945016 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs podName:a1f5a77d-43d5-48bc-a8b5-5db8f1152853 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:11.945010056 +0000 UTC m=+34.246635646 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs") pod "network-metrics-daemon-tt8gq" (UID: "a1f5a77d-43d5-48bc-a8b5-5db8f1152853") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 05:44:56.045897 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:56.045866 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2xnh\" (UniqueName: \"kubernetes.io/projected/529e2d97-b167-4011-a215-02ece8519c00-kube-api-access-k2xnh\") pod \"network-check-target-ljcjx\" (UID: \"529e2d97-b167-4011-a215-02ece8519c00\") " pod="openshift-network-diagnostics/network-check-target-ljcjx" Nov 01 05:44:56.046049 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:56.045995 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 05:44:56.046049 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:56.046009 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 05:44:56.046049 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:56.046018 2586 projected.go:196] Error preparing data for projected volume kube-api-access-k2xnh for pod openshift-network-diagnostics/network-check-target-ljcjx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 05:44:56.046138 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:56.046064 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/529e2d97-b167-4011-a215-02ece8519c00-kube-api-access-k2xnh podName:529e2d97-b167-4011-a215-02ece8519c00 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:12.046052115 +0000 UTC m=+34.347677705 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-k2xnh" (UniqueName: "kubernetes.io/projected/529e2d97-b167-4011-a215-02ece8519c00-kube-api-access-k2xnh") pod "network-check-target-ljcjx" (UID: "529e2d97-b167-4011-a215-02ece8519c00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 05:44:56.172097 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:56.172043 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljcjx" Nov 01 05:44:56.172097 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:56.172090 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:44:56.172357 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:56.172112 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:44:56.172357 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:56.172193 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljcjx" podUID="529e2d97-b167-4011-a215-02ece8519c00" Nov 01 05:44:56.172357 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:56.172320 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tt8gq" podUID="a1f5a77d-43d5-48bc-a8b5-5db8f1152853" Nov 01 05:44:56.172504 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:56.172395 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5l82" podUID="7dc7486d-3188-4bb8-b907-b653caa18660" Nov 01 05:44:56.321729 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:56.321583 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" event={"ID":"21519345-86b8-441e-a395-1e8385cd3732","Type":"ContainerStarted","Data":"31c6cb71d272f6552fda4636fc9a8d7424cb7e9f372fd2c27470dfee91cf1812"} Nov 01 05:44:56.321847 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:56.321791 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" event={"ID":"21519345-86b8-441e-a395-1e8385cd3732","Type":"ContainerStarted","Data":"4a7092913f71d8fffb05dd80364b97e9163e3234d3d41e969b5637f0bbdb370d"} Nov 01 05:44:57.326388 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:57.326353 2586 generic.go:356] "Generic (PLEG): container finished" podID="f5705a40-d3e4-40a8-a561-17c435f388b1" containerID="c13cd4a1ae3808cf52b7c8b377944e16df698151c3bfa767b85c0746892d7cca" exitCode=0 Nov 01 05:44:57.326929 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:57.326441 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7rrq" event={"ID":"f5705a40-d3e4-40a8-a561-17c435f388b1","Type":"ContainerDied","Data":"c13cd4a1ae3808cf52b7c8b377944e16df698151c3bfa767b85c0746892d7cca"} Nov 01 05:44:57.329526 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:57.329506 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" event={"ID":"21519345-86b8-441e-a395-1e8385cd3732","Type":"ContainerStarted","Data":"74ec21a0e877afc590436363d54d018d2adce0dcca1a0db0c057cf714536ae5d"} Nov 01 05:44:57.329579 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:57.329565 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" event={"ID":"21519345-86b8-441e-a395-1e8385cd3732","Type":"ContainerStarted","Data":"476ab85bd212985ec3e6e406fe454e9842fea9c8c793792a107dc2c2e82b6ba2"} Nov 01 05:44:57.329629 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:57.329594 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" event={"ID":"21519345-86b8-441e-a395-1e8385cd3732","Type":"ContainerStarted","Data":"cfb1d43c5937749935657e6ebc408d977fc1d447723166b71f83bf34c9d97614"} Nov 01 05:44:57.329629 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:57.329622 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" event={"ID":"21519345-86b8-441e-a395-1e8385cd3732","Type":"ContainerStarted","Data":"216b4926995f3773e26bb1da326e78ef7aa77e62d847ee8640b3081ee5bea07d"} Nov 01 05:44:58.172830 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:58.172798 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:44:58.173027 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:58.172919 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:44:58.173027 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:58.172935 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tt8gq" podUID="a1f5a77d-43d5-48bc-a8b5-5db8f1152853" Nov 01 05:44:58.173027 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:58.172970 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5l82" podUID="7dc7486d-3188-4bb8-b907-b653caa18660" Nov 01 05:44:58.173027 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:58.172990 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljcjx" Nov 01 05:44:58.173229 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:44:58.173041 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljcjx" podUID="529e2d97-b167-4011-a215-02ece8519c00" Nov 01 05:44:58.334725 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:58.334691 2586 generic.go:356] "Generic (PLEG): container finished" podID="f5705a40-d3e4-40a8-a561-17c435f388b1" containerID="1fa60482427c4deb73102cf745c83c2c89010d2bd8bccad0c328aa12a1feee6c" exitCode=0 Nov 01 05:44:58.335176 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:58.334738 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7rrq" event={"ID":"f5705a40-d3e4-40a8-a561-17c435f388b1","Type":"ContainerDied","Data":"1fa60482427c4deb73102cf745c83c2c89010d2bd8bccad0c328aa12a1feee6c"} Nov 01 05:44:59.339724 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:59.339684 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7rrq" event={"ID":"f5705a40-d3e4-40a8-a561-17c435f388b1","Type":"ContainerStarted","Data":"515e8377b5074568c24a90254828efdc4e60cc4a2380b982a38aa40432dea563"} Nov 01 05:44:59.342692 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:59.342668 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" event={"ID":"21519345-86b8-441e-a395-1e8385cd3732","Type":"ContainerStarted","Data":"c938062236eb7883a25ccab908d6f000f1fad73f9553a7404c3907e5264d2f60"} Nov 01 05:44:59.460110 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:44:59.460064 2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-b7rrq" podStartSLOduration=8.620130057 podStartE2EDuration="21.460048934s" totalImagesPullingTime="12.839918877s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:44:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:44:59.459853188 +0000 UTC m=+21.761478799" watchObservedRunningTime="2026-11-01 05:44:59.460048934 +0000 UTC m=+21.761674548" Nov 01 05:45:00.172786 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:00.172746 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljcjx" Nov 01 05:45:00.172931 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:00.172793 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:45:00.172931 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:00.172760 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:45:00.172931 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:00.172867 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljcjx" podUID="529e2d97-b167-4011-a215-02ece8519c00" Nov 01 05:45:00.173045 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:00.172966 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tt8gq" podUID="a1f5a77d-43d5-48bc-a8b5-5db8f1152853" Nov 01 05:45:00.173075 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:00.173057 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5l82" podUID="7dc7486d-3188-4bb8-b907-b653caa18660" Nov 01 05:45:01.350542 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:01.350504 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" event={"ID":"21519345-86b8-441e-a395-1e8385cd3732","Type":"ContainerStarted","Data":"720df863378a75813830a0c26798811212123bec6ce2a7dd7a3ebddf4708d2b4"} Nov 01 05:45:01.351185 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:01.350809 2586 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:01.351185 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:01.350879 2586 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:01.351185 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:01.350924 2586 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:01.366344 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:01.366186 2586 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:01.366492 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:01.366426 2586 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:01.629169 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:01.629066 2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" podStartSLOduration=7.953190011 podStartE2EDuration="23.629053263s" totalImagesPullingTime="15.675863252s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:01.628667924 +0000 UTC m=+23.930293526" watchObservedRunningTime="2026-11-01 05:45:01.629053263 +0000 UTC m=+23.930678874" Nov 01 05:45:02.172141 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:02.172106 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:45:02.172416 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:02.172148 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljcjx" Nov 01 05:45:02.172416 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:02.172242 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5l82" podUID="7dc7486d-3188-4bb8-b907-b653caa18660" Nov 01 05:45:02.172416 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:02.172295 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljcjx" podUID="529e2d97-b167-4011-a215-02ece8519c00" Nov 01 05:45:02.172416 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:02.172335 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:45:02.172416 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:02.172406 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tt8gq" podUID="a1f5a77d-43d5-48bc-a8b5-5db8f1152853" Nov 01 05:45:03.377284 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:03.377247 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r5l82"] Nov 01 05:45:03.377765 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:03.377370 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:45:03.377765 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:03.377494 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5l82" podUID="7dc7486d-3188-4bb8-b907-b653caa18660" Nov 01 05:45:03.377872 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:03.377829 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ljcjx"] Nov 01 05:45:03.377920 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:03.377913 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljcjx" Nov 01 05:45:03.378009 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:03.377987 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljcjx" podUID="529e2d97-b167-4011-a215-02ece8519c00" Nov 01 05:45:03.426627 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:03.426597 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tt8gq"] Nov 01 05:45:03.426759 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:03.426703 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:45:03.427181 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:03.427149 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tt8gq" podUID="a1f5a77d-43d5-48bc-a8b5-5db8f1152853" Nov 01 05:45:05.172411 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:05.172378 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljcjx" Nov 01 05:45:05.172841 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:05.172438 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:45:05.172841 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:05.172441 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:45:05.172841 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:05.172546 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tt8gq" podUID="a1f5a77d-43d5-48bc-a8b5-5db8f1152853" Nov 01 05:45:05.172841 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:05.172598 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5l82" podUID="7dc7486d-3188-4bb8-b907-b653caa18660" Nov 01 05:45:05.172841 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:05.172657 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljcjx" podUID="529e2d97-b167-4011-a215-02ece8519c00" Nov 01 05:45:06.726071 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:06.726036 2586 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-n7mks" Nov 01 05:45:06.726566 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:06.726202 2586 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 05:45:06.726652 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:06.726636 2586 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-n7mks" Nov 01 05:45:07.171677 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:07.171646 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:45:07.171677 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:07.171665 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljcjx" Nov 01 05:45:07.171918 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:07.171690 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:45:07.171918 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:07.171800 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5l82" podUID="7dc7486d-3188-4bb8-b907-b653caa18660" Nov 01 05:45:07.171918 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:07.171908 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljcjx" podUID="529e2d97-b167-4011-a215-02ece8519c00" Nov 01 05:45:07.172060 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:07.171980 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tt8gq" podUID="a1f5a77d-43d5-48bc-a8b5-5db8f1152853" Nov 01 05:45:09.171862 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.171825 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:45:09.171862 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.171850 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:45:09.172400 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.171825 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljcjx" Nov 01 05:45:09.172400 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:09.171951 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5l82" podUID="7dc7486d-3188-4bb8-b907-b653caa18660" Nov 01 05:45:09.172400 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:09.172001 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljcjx" podUID="529e2d97-b167-4011-a215-02ece8519c00" Nov 01 05:45:09.172400 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:09.172072 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tt8gq" podUID="a1f5a77d-43d5-48bc-a8b5-5db8f1152853" Nov 01 05:45:09.519097 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.519069 2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-195.us-west-1.compute.internal" event="NodeReady" Nov 01 05:45:09.519260 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.519220 2586 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 01 05:45:09.605822 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.605792 2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-b2kmh"] Nov 01 05:45:09.678042 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.678008 2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ls8v2"] Nov 01 05:45:09.678205 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.678132 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-b2kmh" Nov 01 05:45:09.692614 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.692594 2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-b2659"] Nov 01 05:45:09.692737 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.692718 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ls8v2" Nov 01 05:45:09.707959 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.707934 2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-9j8j4"] Nov 01 05:45:09.708079 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.708049 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-b2659" Nov 01 05:45:09.722742 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.722722 2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6c5b5d5bd5-776bc"] Nov 01 05:45:09.722849 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.722833 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9j8j4" Nov 01 05:45:09.730257 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.730239 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 01 05:45:09.730358 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.730289 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 01 05:45:09.736954 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.736929 2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95sds"] Nov 01 05:45:09.737088 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.737053 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:09.759152 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.759126 2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-54lc7"] Nov 01 05:45:09.759303 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.759259 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95sds" Nov 01 05:45:09.774955 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.774910 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 01 05:45:09.775050 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.774991 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-9g7mn\"" Nov 01 05:45:09.781727 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.781710 2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flt78"] Nov 01 05:45:09.781836 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.781822 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-54lc7" Nov 01 05:45:09.789759 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.789742 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 01 05:45:09.794174 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.794152 2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-njjn4"] Nov 01 05:45:09.794286 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.794273 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flt78" Nov 01 05:45:09.809597 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.809572 2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5f55979cbb-ngg8z"] Nov 01 05:45:09.809733 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.809691 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" Nov 01 05:45:09.829424 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.829400 2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-7grdp"] Nov 01 05:45:09.829581 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.829553 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:09.830759 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.830735 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 05:45:09.830759 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.830740 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 01 05:45:09.831070 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.830769 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 01 05:45:09.831070 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.830775 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 01 05:45:09.831070 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.830759 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 01 05:45:09.831070 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.830803 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 01 05:45:09.831070 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.830838 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 01 05:45:09.831225 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.831118 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 05:45:09.831225 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.831214 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 01 05:45:09.833388 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.833374 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 01 05:45:09.833576 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.833563 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 01 05:45:09.840274 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.840247 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/109327c0-f8f8-4dde-9cfe-489bf5a7eaad-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-95sds\" (UID: \"109327c0-f8f8-4dde-9cfe-489bf5a7eaad\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95sds" Nov 01 05:45:09.840375 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.840311 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6r8g\" (UniqueName: \"kubernetes.io/projected/31894c07-ba1e-4865-9481-58555f919bcc-kube-api-access-t6r8g\") pod \"kas-connection-checker-685895c7c9-ls8v2\" (UID: \"31894c07-ba1e-4865-9481-58555f919bcc\") " pod="kube-system/kas-connection-checker-685895c7c9-ls8v2" Nov 01 05:45:09.840375 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.840361 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swfdf\" (UniqueName: \"kubernetes.io/projected/5e70277c-fae5-4e5a-b3d0-743d7485d2b3-kube-api-access-swfdf\") pod \"kas-connection-checker-685895c7c9-b2659\" (UID: \"5e70277c-fae5-4e5a-b3d0-743d7485d2b3\") " pod="kube-system/kas-connection-checker-685895c7c9-b2659" Nov 01 05:45:09.840458 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.840419 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-service-ca-bundle\") pod \"router-default-6c5b5d5bd5-776bc\" (UID: \"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78\") " pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:09.840458 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.840439 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dc9422c2-b041-4f6b-8495-961a9571c1e4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-9j8j4\" (UID: \"dc9422c2-b041-4f6b-8495-961a9571c1e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9j8j4" Nov 01 05:45:09.840582 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.840515 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-default-certificate\") pod \"router-default-6c5b5d5bd5-776bc\" (UID: \"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78\") " pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:09.840582 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.840547 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-stats-auth\") pod \"router-default-6c5b5d5bd5-776bc\" (UID: \"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78\") " pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:09.840582 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.840575 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-metrics-certs\") pod \"router-default-6c5b5d5bd5-776bc\" (UID: \"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78\") " pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:09.840692 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.840641 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7854\" (UniqueName: \"kubernetes.io/projected/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-kube-api-access-b7854\") pod \"router-default-6c5b5d5bd5-776bc\" (UID: \"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78\") " pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:09.840692 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.840658 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/109327c0-f8f8-4dde-9cfe-489bf5a7eaad-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-95sds\" (UID: \"109327c0-f8f8-4dde-9cfe-489bf5a7eaad\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95sds" Nov 01 05:45:09.840692 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.840677 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2nbg\" (UniqueName: \"kubernetes.io/projected/1cc9d6f4-202c-45bb-8a4c-4b8e936874de-kube-api-access-n2nbg\") pod \"kas-connection-checker-685895c7c9-b2kmh\" (UID: \"1cc9d6f4-202c-45bb-8a4c-4b8e936874de\") " pod="kube-system/kas-connection-checker-685895c7c9-b2kmh" Nov 01 05:45:09.840799 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.840695 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc9422c2-b041-4f6b-8495-961a9571c1e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9j8j4\" (UID: \"dc9422c2-b041-4f6b-8495-961a9571c1e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9j8j4" Nov 01 05:45:09.840799 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.840731 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j64kh\" (UniqueName: \"kubernetes.io/projected/109327c0-f8f8-4dde-9cfe-489bf5a7eaad-kube-api-access-j64kh\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-95sds\" (UID: \"109327c0-f8f8-4dde-9cfe-489bf5a7eaad\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95sds" Nov 01 05:45:09.840864 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.840806 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsp9k\" (UniqueName: \"kubernetes.io/projected/dc9422c2-b041-4f6b-8495-961a9571c1e4-kube-api-access-jsp9k\") pod \"cluster-monitoring-operator-884576c57-9j8j4\" (UID: \"dc9422c2-b041-4f6b-8495-961a9571c1e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9j8j4" Nov 01 05:45:09.841309 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.841289 2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-vr978"] Nov 01 05:45:09.841418 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.841404 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-7grdp" Nov 01 05:45:09.851075 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.851059 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 01 05:45:09.851075 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.851070 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 01 05:45:09.851195 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.851078 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 01 05:45:09.861569 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.861527 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 01 05:45:09.861569 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.861527 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 01 05:45:09.861569 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.861566 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 01 05:45:09.865715 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.865699 2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-r8n4b"] Nov 01 05:45:09.865822 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.865810 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vr978" Nov 01 05:45:09.880327 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.880307 2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-847d56945f-lt2tl"] Nov 01 05:45:09.880522 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.880427 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-r8n4b" Nov 01 05:45:09.881042 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.881014 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 01 05:45:09.881135 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.881092 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 01 05:45:09.881135 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.881109 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 01 05:45:09.881231 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.881145 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 01 05:45:09.881231 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.881142 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-qxt94\"" Nov 01 05:45:09.881231 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.881139 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 01 05:45:09.881231 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.881092 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 01 05:45:09.881231 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.881200 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7s2ht\"" Nov 01 05:45:09.881430 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.881290 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-g592c\"" Nov 01 05:45:09.881430 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.881341 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-6j8z2\"" Nov 01 05:45:09.881430 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.881356 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 01 05:45:09.881585 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.881501 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 01 05:45:09.881585 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.881517 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p4xg8\"" Nov 01 05:45:09.881585 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.881529 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 01 05:45:09.881585 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.881531 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-mzdd9\"" Nov 01 05:45:09.893102 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.893083 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 01 05:45:09.893685 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.893671 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-vbztt\"" Nov 01 05:45:09.900234 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.900215 2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-pgxkq"] Nov 01 05:45:09.900377 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.900361 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:09.922474 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.922413 2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hxw5f"] Nov 01 05:45:09.922618 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.922542 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-pgxkq" Nov 01 05:45:09.932839 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.932799 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 01 05:45:09.934216 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.934196 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 01 05:45:09.941407 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.941390 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/580bebeb-9495-4541-8d1d-3bf6fef1a061-config\") pod \"console-operator-7cd56b8dbc-njjn4\" (UID: \"580bebeb-9495-4541-8d1d-3bf6fef1a061\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" Nov 01 05:45:09.941474 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.941423 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6r8g\" (UniqueName: \"kubernetes.io/projected/31894c07-ba1e-4865-9481-58555f919bcc-kube-api-access-t6r8g\") pod \"kas-connection-checker-685895c7c9-ls8v2\" (UID: \"31894c07-ba1e-4865-9481-58555f919bcc\") " pod="kube-system/kas-connection-checker-685895c7c9-ls8v2" Nov 01 05:45:09.941474 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.941439 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swfdf\" (UniqueName: \"kubernetes.io/projected/5e70277c-fae5-4e5a-b3d0-743d7485d2b3-kube-api-access-swfdf\") pod \"kas-connection-checker-685895c7c9-b2659\" (UID: \"5e70277c-fae5-4e5a-b3d0-743d7485d2b3\") " pod="kube-system/kas-connection-checker-685895c7c9-b2659" Nov 01 05:45:09.941474 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.941458 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/293a6ef2-f0c9-4390-ac82-028792a3b64d-serving-cert\") pod \"service-ca-operator-544d749c8c-vr978\" (UID: \"293a6ef2-f0c9-4390-ac82-028792a3b64d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vr978" Nov 01 05:45:09.941638 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.941619 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f9ccbc38-43f9-4396-bb59-2f746b3d639d-installation-pull-secrets\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:09.941772 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.941750 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/109327c0-f8f8-4dde-9cfe-489bf5a7eaad-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-95sds\" (UID: \"109327c0-f8f8-4dde-9cfe-489bf5a7eaad\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95sds" Nov 01 05:45:09.941817 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.941791 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-flt78\" (UID: \"320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flt78" Nov 01 05:45:09.941848 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.941823 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-service-ca-bundle\") pod \"router-default-6c5b5d5bd5-776bc\" (UID: \"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78\") " pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:09.941890 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.941857 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/293a6ef2-f0c9-4390-ac82-028792a3b64d-config\") pod \"service-ca-operator-544d749c8c-vr978\" (UID: \"293a6ef2-f0c9-4390-ac82-028792a3b64d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vr978" Nov 01 05:45:09.941944 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.941899 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-metrics-certs\") pod \"router-default-6c5b5d5bd5-776bc\" (UID: \"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78\") " pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:09.941944 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.941925 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f9ccbc38-43f9-4396-bb59-2f746b3d639d-registry-certificates\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:09.942040 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.941967 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-registry-tls\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:09.942040 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:09.941988 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-service-ca-bundle podName:fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:10.441966702 +0000 UTC m=+32.743592336 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-service-ca-bundle") pod "router-default-6c5b5d5bd5-776bc" (UID: "fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78") : configmap references non-existent config key: service-ca.crt Nov 01 05:45:09.942040 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:09.942012 2586 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 05:45:09.942186 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.942053 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc9422c2-b041-4f6b-8495-961a9571c1e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9j8j4\" (UID: \"dc9422c2-b041-4f6b-8495-961a9571c1e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9j8j4" Nov 01 05:45:09.942186 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:09.942059 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-metrics-certs podName:fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:10.442045855 +0000 UTC m=+32.743671448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-metrics-certs") pod "router-default-6c5b5d5bd5-776bc" (UID: "fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78") : secret "router-metrics-certs-default" not found Nov 01 05:45:09.942186 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.942093 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f9ccbc38-43f9-4396-bb59-2f746b3d639d-trusted-ca\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:09.942186 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.942120 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/580bebeb-9495-4541-8d1d-3bf6fef1a061-serving-cert\") pod \"console-operator-7cd56b8dbc-njjn4\" (UID: \"580bebeb-9495-4541-8d1d-3bf6fef1a061\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" Nov 01 05:45:09.942186 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:09.942130 2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 05:45:09.942186 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.942140 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j64kh\" (UniqueName: \"kubernetes.io/projected/109327c0-f8f8-4dde-9cfe-489bf5a7eaad-kube-api-access-j64kh\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-95sds\" (UID: \"109327c0-f8f8-4dde-9cfe-489bf5a7eaad\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95sds" Nov 01 05:45:09.942186 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:09.942172 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc9422c2-b041-4f6b-8495-961a9571c1e4-cluster-monitoring-operator-tls podName:dc9422c2-b041-4f6b-8495-961a9571c1e4 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:10.442159239 +0000 UTC m=+32.743784851 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dc9422c2-b041-4f6b-8495-961a9571c1e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9j8j4" (UID: "dc9422c2-b041-4f6b-8495-961a9571c1e4") : secret "cluster-monitoring-operator-tls" not found Nov 01 05:45:09.942504 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.942191 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsp9k\" (UniqueName: \"kubernetes.io/projected/dc9422c2-b041-4f6b-8495-961a9571c1e4-kube-api-access-jsp9k\") pod \"cluster-monitoring-operator-884576c57-9j8j4\" (UID: \"dc9422c2-b041-4f6b-8495-961a9571c1e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9j8j4" Nov 01 05:45:09.942504 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.942218 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/109327c0-f8f8-4dde-9cfe-489bf5a7eaad-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-95sds\" (UID: \"109327c0-f8f8-4dde-9cfe-489bf5a7eaad\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95sds" Nov 01 05:45:09.942504 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.942246 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk8zm\" (UniqueName: \"kubernetes.io/projected/580bebeb-9495-4541-8d1d-3bf6fef1a061-kube-api-access-mk8zm\") pod \"console-operator-7cd56b8dbc-njjn4\" (UID: \"580bebeb-9495-4541-8d1d-3bf6fef1a061\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" Nov 01 05:45:09.942504 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.942288 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-flt78\" (UID: \"320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flt78" Nov 01 05:45:09.942504 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.942313 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f9ccbc38-43f9-4396-bb59-2f746b3d639d-image-registry-private-configuration\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:09.942504 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.942375 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f9ccbc38-43f9-4396-bb59-2f746b3d639d-ca-trust-extracted\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:09.942504 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.942418 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgrrf\" (UniqueName: \"kubernetes.io/projected/f4051a42-5120-4b67-b904-5db6b4897e07-kube-api-access-vgrrf\") pod \"volume-data-source-validator-58c9f5f46d-54lc7\" (UID: \"f4051a42-5120-4b67-b904-5db6b4897e07\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-54lc7" Nov 01 05:45:09.942504 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.942451 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dc9422c2-b041-4f6b-8495-961a9571c1e4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-9j8j4\" (UID: \"dc9422c2-b041-4f6b-8495-961a9571c1e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9j8j4" Nov 01 05:45:09.942818 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.942513 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lmfd\" (UniqueName: \"kubernetes.io/projected/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-kube-api-access-2lmfd\") pod \"cluster-samples-operator-6ddb6c8b94-flt78\" (UID: \"320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flt78" Nov 01 05:45:09.942818 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.942552 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-stats-auth\") pod \"router-default-6c5b5d5bd5-776bc\" (UID: \"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78\") " pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:09.942818 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.942577 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pdtj\" (UniqueName: \"kubernetes.io/projected/293a6ef2-f0c9-4390-ac82-028792a3b64d-kube-api-access-6pdtj\") pod \"service-ca-operator-544d749c8c-vr978\" (UID: \"293a6ef2-f0c9-4390-ac82-028792a3b64d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vr978" Nov 01 05:45:09.942818 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.942628 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-default-certificate\") pod \"router-default-6c5b5d5bd5-776bc\" (UID: \"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78\") " pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:09.942818 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.942689 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/375debb4-cfe1-44ab-907b-0a34a031c2ea-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7grdp\" (UID: \"375debb4-cfe1-44ab-907b-0a34a031c2ea\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7grdp" Nov 01 05:45:09.942818 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.942714 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-bound-sa-token\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:09.942818 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.942745 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7854\" (UniqueName: \"kubernetes.io/projected/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-kube-api-access-b7854\") pod \"router-default-6c5b5d5bd5-776bc\" (UID: \"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78\") " pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:09.942818 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.942788 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2nbg\" (UniqueName: \"kubernetes.io/projected/1cc9d6f4-202c-45bb-8a4c-4b8e936874de-kube-api-access-n2nbg\") pod \"kas-connection-checker-685895c7c9-b2kmh\" (UID: \"1cc9d6f4-202c-45bb-8a4c-4b8e936874de\") " pod="kube-system/kas-connection-checker-685895c7c9-b2kmh" Nov 01 05:45:09.942818 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.942818 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/580bebeb-9495-4541-8d1d-3bf6fef1a061-trusted-ca\") pod \"console-operator-7cd56b8dbc-njjn4\" (UID: \"580bebeb-9495-4541-8d1d-3bf6fef1a061\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" Nov 01 05:45:09.943160 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.942821 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/109327c0-f8f8-4dde-9cfe-489bf5a7eaad-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-95sds\" (UID: \"109327c0-f8f8-4dde-9cfe-489bf5a7eaad\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95sds" Nov 01 05:45:09.943160 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.942844 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqqbs\" (UniqueName: \"kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-kube-api-access-vqqbs\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:09.943160 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.942878 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/375debb4-cfe1-44ab-907b-0a34a031c2ea-nginx-conf\") pod \"networking-console-plugin-86c478bb88-7grdp\" (UID: \"375debb4-cfe1-44ab-907b-0a34a031c2ea\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7grdp" Nov 01 05:45:09.943344 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.943235 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dc9422c2-b041-4f6b-8495-961a9571c1e4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-9j8j4\" (UID: \"dc9422c2-b041-4f6b-8495-961a9571c1e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9j8j4" Nov 01 05:45:09.943607 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.943589 2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dwklx"] Nov 01 05:45:09.943725 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.943707 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hxw5f" Nov 01 05:45:09.945696 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.945672 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/109327c0-f8f8-4dde-9cfe-489bf5a7eaad-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-95sds\" (UID: \"109327c0-f8f8-4dde-9cfe-489bf5a7eaad\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95sds" Nov 01 05:45:09.945765 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.945730 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-default-certificate\") pod \"router-default-6c5b5d5bd5-776bc\" (UID: \"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78\") " pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:09.945811 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.945769 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-stats-auth\") pod \"router-default-6c5b5d5bd5-776bc\" (UID: \"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78\") " pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:09.965944 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.965922 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ls8v2"] Nov 01 05:45:09.966031 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.965965 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-b2659"] Nov 01 05:45:09.966031 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.965990 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-b2kmh"] Nov 01 05:45:09.966031 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.966012 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-9j8j4"] Nov 01 05:45:09.966124 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.966032 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-vr978"] Nov 01 05:45:09.966124 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.966035 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dwklx" Nov 01 05:45:09.966124 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.966056 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-njjn4"] Nov 01 05:45:09.966124 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.966077 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flt78"] Nov 01 05:45:09.966124 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.966104 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6c5b5d5bd5-776bc"] Nov 01 05:45:09.966124 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.966123 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dwklx"] Nov 01 05:45:09.966295 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.966142 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-r8n4b"] Nov 01 05:45:09.966295 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.966161 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-7grdp"] Nov 01 05:45:09.966295 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.966188 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-54lc7"] Nov 01 05:45:09.966295 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.966219 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hxw5f"] Nov 01 05:45:09.966295 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.966251 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-pgxkq"] Nov 01 05:45:09.966295 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.966273 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95sds"] Nov 01 05:45:09.966295 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.966295 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-847d56945f-lt2tl"] Nov 01 05:45:09.966584 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.966314 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f55979cbb-ngg8z"] Nov 01 05:45:09.974961 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.974942 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 01 05:45:09.975063 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.974999 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 01 05:45:09.975063 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:09.975024 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-wvmhq\"" Nov 01 05:45:10.007415 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.005128 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j64kh\" (UniqueName: \"kubernetes.io/projected/109327c0-f8f8-4dde-9cfe-489bf5a7eaad-kube-api-access-j64kh\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-95sds\" (UID: \"109327c0-f8f8-4dde-9cfe-489bf5a7eaad\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95sds" Nov 01 05:45:10.007415 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.005168 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6r8g\" (UniqueName: \"kubernetes.io/projected/31894c07-ba1e-4865-9481-58555f919bcc-kube-api-access-t6r8g\") pod \"kas-connection-checker-685895c7c9-ls8v2\" (UID: \"31894c07-ba1e-4865-9481-58555f919bcc\") " pod="kube-system/kas-connection-checker-685895c7c9-ls8v2" Nov 01 05:45:10.007415 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.005315 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsp9k\" (UniqueName: \"kubernetes.io/projected/dc9422c2-b041-4f6b-8495-961a9571c1e4-kube-api-access-jsp9k\") pod \"cluster-monitoring-operator-884576c57-9j8j4\" (UID: \"dc9422c2-b041-4f6b-8495-961a9571c1e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9j8j4" Nov 01 05:45:10.007922 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.007897 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swfdf\" (UniqueName: \"kubernetes.io/projected/5e70277c-fae5-4e5a-b3d0-743d7485d2b3-kube-api-access-swfdf\") pod \"kas-connection-checker-685895c7c9-b2659\" (UID: \"5e70277c-fae5-4e5a-b3d0-743d7485d2b3\") " pod="kube-system/kas-connection-checker-685895c7c9-b2659" Nov 01 05:45:10.008790 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.008768 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7854\" (UniqueName: \"kubernetes.io/projected/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-kube-api-access-b7854\") pod \"router-default-6c5b5d5bd5-776bc\" (UID: \"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78\") " pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:10.008910 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.008875 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2nbg\" (UniqueName: \"kubernetes.io/projected/1cc9d6f4-202c-45bb-8a4c-4b8e936874de-kube-api-access-n2nbg\") pod \"kas-connection-checker-685895c7c9-b2kmh\" (UID: \"1cc9d6f4-202c-45bb-8a4c-4b8e936874de\") " pod="kube-system/kas-connection-checker-685895c7c9-b2kmh" Nov 01 05:45:10.015568 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.015544 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ls8v2" Nov 01 05:45:10.029888 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.029823 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-b2659" Nov 01 05:45:10.031829 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.031799 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 05:45:10.031829 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.031799 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 05:45:10.031973 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.031830 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 01 05:45:10.031973 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.031804 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 01 05:45:10.031973 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.031850 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 01 05:45:10.031973 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.031806 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 05:45:10.031973 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.031869 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 05:45:10.031973 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.031893 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 05:45:10.031973 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.031870 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 01 05:45:10.031973 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.031877 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 05:45:10.032339 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.031985 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 05:45:10.032657 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.032643 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 01 05:45:10.043985 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.043960 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f9ccbc38-43f9-4396-bb59-2f746b3d639d-ca-trust-extracted\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:10.044099 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.043999 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7729d415-70e5-4be3-a9cc-bbd13e065c71-registry-certificates\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:10.044099 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.044022 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7729d415-70e5-4be3-a9cc-bbd13e065c71-installation-pull-secrets\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:10.044099 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.044087 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgrrf\" (UniqueName: \"kubernetes.io/projected/f4051a42-5120-4b67-b904-5db6b4897e07-kube-api-access-vgrrf\") pod \"volume-data-source-validator-58c9f5f46d-54lc7\" (UID: \"f4051a42-5120-4b67-b904-5db6b4897e07\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-54lc7" Nov 01 05:45:10.044293 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.044273 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f98900-ff17-447f-8429-60c3e47fc7d1-tmp-dir\") pod \"dns-default-hxw5f\" (UID: \"72f98900-ff17-447f-8429-60c3e47fc7d1\") " pod="openshift-dns/dns-default-hxw5f" Nov 01 05:45:10.044346 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.044321 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hksk\" (UniqueName: \"kubernetes.io/projected/cbc9c127-21a0-4020-a093-d02b4061db5a-kube-api-access-6hksk\") pod \"insights-operator-7c4986d449-r8n4b\" (UID: \"cbc9c127-21a0-4020-a093-d02b4061db5a\") " pod="openshift-insights/insights-operator-7c4986d449-r8n4b" Nov 01 05:45:10.044400 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.044363 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lmfd\" (UniqueName: \"kubernetes.io/projected/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-kube-api-access-2lmfd\") pod \"cluster-samples-operator-6ddb6c8b94-flt78\" (UID: \"320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flt78" Nov 01 05:45:10.044446 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.044395 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f9ccbc38-43f9-4396-bb59-2f746b3d639d-ca-trust-extracted\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:10.044446 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.044426 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzj6t\" (UniqueName: \"kubernetes.io/projected/1b19c3b0-887e-4e27-bac3-028003f1aac5-kube-api-access-fzj6t\") pod \"ingress-canary-dwklx\" (UID: \"1b19c3b0-887e-4e27-bac3-028003f1aac5\") " pod="openshift-ingress-canary/ingress-canary-dwklx" Nov 01 05:45:10.044536 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.044465 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pdtj\" (UniqueName: \"kubernetes.io/projected/293a6ef2-f0c9-4390-ac82-028792a3b64d-kube-api-access-6pdtj\") pod \"service-ca-operator-544d749c8c-vr978\" (UID: \"293a6ef2-f0c9-4390-ac82-028792a3b64d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vr978" Nov 01 05:45:10.044569 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.044531 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/375debb4-cfe1-44ab-907b-0a34a031c2ea-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7grdp\" (UID: \"375debb4-cfe1-44ab-907b-0a34a031c2ea\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7grdp" Nov 01 05:45:10.044569 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.044555 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-bound-sa-token\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:10.044660 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.044572 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5rfj\" (UniqueName: \"kubernetes.io/projected/72f98900-ff17-447f-8429-60c3e47fc7d1-kube-api-access-r5rfj\") pod \"dns-default-hxw5f\" (UID: \"72f98900-ff17-447f-8429-60c3e47fc7d1\") " pod="openshift-dns/dns-default-hxw5f" Nov 01 05:45:10.044660 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.044596 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbc9c127-21a0-4020-a093-d02b4061db5a-service-ca-bundle\") pod \"insights-operator-7c4986d449-r8n4b\" (UID: \"cbc9c127-21a0-4020-a093-d02b4061db5a\") " pod="openshift-insights/insights-operator-7c4986d449-r8n4b" Nov 01 05:45:10.044660 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.044640 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/580bebeb-9495-4541-8d1d-3bf6fef1a061-trusted-ca\") pod \"console-operator-7cd56b8dbc-njjn4\" (UID: \"580bebeb-9495-4541-8d1d-3bf6fef1a061\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" Nov 01 05:45:10.044660 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:10.044652 2586 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 05:45:10.044824 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.044677 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqqbs\" (UniqueName: \"kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-kube-api-access-vqqbs\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:10.044824 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.044753 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72qwb\" (UniqueName: \"kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-kube-api-access-72qwb\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:10.044824 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:10.044773 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/375debb4-cfe1-44ab-907b-0a34a031c2ea-networking-console-plugin-cert podName:375debb4-cfe1-44ab-907b-0a34a031c2ea nodeName:}" failed. No retries permitted until 2026-11-01 05:45:10.544751942 +0000 UTC m=+32.846377547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/375debb4-cfe1-44ab-907b-0a34a031c2ea-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7grdp" (UID: "375debb4-cfe1-44ab-907b-0a34a031c2ea") : secret "networking-console-plugin-cert" not found Nov 01 05:45:10.044824 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.044796 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/375debb4-cfe1-44ab-907b-0a34a031c2ea-nginx-conf\") pod \"networking-console-plugin-86c478bb88-7grdp\" (UID: \"375debb4-cfe1-44ab-907b-0a34a031c2ea\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7grdp" Nov 01 05:45:10.044824 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.044815 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7729d415-70e5-4be3-a9cc-bbd13e065c71-trusted-ca\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:10.045082 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.044839 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/580bebeb-9495-4541-8d1d-3bf6fef1a061-config\") pod \"console-operator-7cd56b8dbc-njjn4\" (UID: \"580bebeb-9495-4541-8d1d-3bf6fef1a061\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" Nov 01 05:45:10.045082 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.044878 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b19c3b0-887e-4e27-bac3-028003f1aac5-cert\") pod \"ingress-canary-dwklx\" (UID: \"1b19c3b0-887e-4e27-bac3-028003f1aac5\") " pod="openshift-ingress-canary/ingress-canary-dwklx" Nov 01 05:45:10.045082 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.044900 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7729d415-70e5-4be3-a9cc-bbd13e065c71-image-registry-private-configuration\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:10.045082 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.044921 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/293a6ef2-f0c9-4390-ac82-028792a3b64d-serving-cert\") pod \"service-ca-operator-544d749c8c-vr978\" (UID: \"293a6ef2-f0c9-4390-ac82-028792a3b64d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vr978" Nov 01 05:45:10.045082 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.044938 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f9ccbc38-43f9-4396-bb59-2f746b3d639d-installation-pull-secrets\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:10.045082 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.044986 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7729d415-70e5-4be3-a9cc-bbd13e065c71-ca-trust-extracted\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:10.045082 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.045027 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cbc9c127-21a0-4020-a093-d02b4061db5a-serving-cert\") pod \"insights-operator-7c4986d449-r8n4b\" (UID: \"cbc9c127-21a0-4020-a093-d02b4061db5a\") " pod="openshift-insights/insights-operator-7c4986d449-r8n4b" Nov 01 05:45:10.045082 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.045059 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-flt78\" (UID: \"320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flt78" Nov 01 05:45:10.045436 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.045120 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/293a6ef2-f0c9-4390-ac82-028792a3b64d-config\") pod \"service-ca-operator-544d749c8c-vr978\" (UID: \"293a6ef2-f0c9-4390-ac82-028792a3b64d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vr978" Nov 01 05:45:10.045436 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:10.045132 2586 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 05:45:10.045436 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:10.045180 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-samples-operator-tls podName:320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:10.545161566 +0000 UTC m=+32.846787174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-flt78" (UID: "320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2") : secret "samples-operator-tls" not found Nov 01 05:45:10.045436 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.045199 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f9ccbc38-43f9-4396-bb59-2f746b3d639d-registry-certificates\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:10.045436 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.045233 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-registry-tls\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:10.045436 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.045262 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbc9c127-21a0-4020-a093-d02b4061db5a-tmp\") pod \"insights-operator-7c4986d449-r8n4b\" (UID: \"cbc9c127-21a0-4020-a093-d02b4061db5a\") " pod="openshift-insights/insights-operator-7c4986d449-r8n4b" Nov 01 05:45:10.045436 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.045288 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-bound-sa-token\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:10.045436 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.045321 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-registry-tls\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:10.045436 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.045364 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f9ccbc38-43f9-4396-bb59-2f746b3d639d-trusted-ca\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:10.045436 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.045393 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cbc9c127-21a0-4020-a093-d02b4061db5a-snapshots\") pod \"insights-operator-7c4986d449-r8n4b\" (UID: \"cbc9c127-21a0-4020-a093-d02b4061db5a\") " pod="openshift-insights/insights-operator-7c4986d449-r8n4b" Nov 01 05:45:10.045436 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.045421 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/580bebeb-9495-4541-8d1d-3bf6fef1a061-serving-cert\") pod \"console-operator-7cd56b8dbc-njjn4\" (UID: \"580bebeb-9495-4541-8d1d-3bf6fef1a061\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" Nov 01 05:45:10.045966 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.045449 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdc27\" (UniqueName: \"kubernetes.io/projected/f8238f6d-c793-4f76-b63b-d2d50f9a585d-kube-api-access-cdc27\") pod \"network-check-source-6f78d59f99-pgxkq\" (UID: \"f8238f6d-c793-4f76-b63b-d2d50f9a585d\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-pgxkq" Nov 01 05:45:10.045966 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.045499 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk8zm\" (UniqueName: \"kubernetes.io/projected/580bebeb-9495-4541-8d1d-3bf6fef1a061-kube-api-access-mk8zm\") pod \"console-operator-7cd56b8dbc-njjn4\" (UID: \"580bebeb-9495-4541-8d1d-3bf6fef1a061\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" Nov 01 05:45:10.045966 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.045524 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/72f98900-ff17-447f-8429-60c3e47fc7d1-config-volume\") pod \"dns-default-hxw5f\" (UID: \"72f98900-ff17-447f-8429-60c3e47fc7d1\") " pod="openshift-dns/dns-default-hxw5f" Nov 01 05:45:10.045966 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.045546 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/375debb4-cfe1-44ab-907b-0a34a031c2ea-nginx-conf\") pod \"networking-console-plugin-86c478bb88-7grdp\" (UID: \"375debb4-cfe1-44ab-907b-0a34a031c2ea\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7grdp" Nov 01 05:45:10.045966 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.045547 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/580bebeb-9495-4541-8d1d-3bf6fef1a061-config\") pod \"console-operator-7cd56b8dbc-njjn4\" (UID: \"580bebeb-9495-4541-8d1d-3bf6fef1a061\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" Nov 01 05:45:10.045966 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.045550 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72f98900-ff17-447f-8429-60c3e47fc7d1-metrics-tls\") pod \"dns-default-hxw5f\" (UID: \"72f98900-ff17-447f-8429-60c3e47fc7d1\") " pod="openshift-dns/dns-default-hxw5f" Nov 01 05:45:10.045966 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:10.045644 2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 05:45:10.045966 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:10.045660 2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f55979cbb-ngg8z: secret "image-registry-tls" not found Nov 01 05:45:10.045966 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.045670 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/580bebeb-9495-4541-8d1d-3bf6fef1a061-trusted-ca\") pod \"console-operator-7cd56b8dbc-njjn4\" (UID: \"580bebeb-9495-4541-8d1d-3bf6fef1a061\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" Nov 01 05:45:10.045966 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.045709 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/293a6ef2-f0c9-4390-ac82-028792a3b64d-config\") pod \"service-ca-operator-544d749c8c-vr978\" (UID: \"293a6ef2-f0c9-4390-ac82-028792a3b64d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vr978" Nov 01 05:45:10.045966 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:10.045732 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-registry-tls podName:f9ccbc38-43f9-4396-bb59-2f746b3d639d nodeName:}" failed. No retries permitted until 2026-11-01 05:45:10.545717956 +0000 UTC m=+32.847343563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-registry-tls") pod "image-registry-5f55979cbb-ngg8z" (UID: "f9ccbc38-43f9-4396-bb59-2f746b3d639d") : secret "image-registry-tls" not found Nov 01 05:45:10.045966 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.045772 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f9ccbc38-43f9-4396-bb59-2f746b3d639d-registry-certificates\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:10.045966 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.045775 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbc9c127-21a0-4020-a093-d02b4061db5a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-r8n4b\" (UID: \"cbc9c127-21a0-4020-a093-d02b4061db5a\") " pod="openshift-insights/insights-operator-7c4986d449-r8n4b" Nov 01 05:45:10.045966 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.045846 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-flt78\" (UID: \"320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flt78" Nov 01 05:45:10.045966 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.045884 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f9ccbc38-43f9-4396-bb59-2f746b3d639d-image-registry-private-configuration\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:10.047634 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.047432 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f9ccbc38-43f9-4396-bb59-2f746b3d639d-trusted-ca\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:10.047634 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.047468 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-flt78\" (UID: \"320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flt78" Nov 01 05:45:10.048072 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.047997 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f9ccbc38-43f9-4396-bb59-2f746b3d639d-installation-pull-secrets\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:10.048116 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.048089 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/580bebeb-9495-4541-8d1d-3bf6fef1a061-serving-cert\") pod \"console-operator-7cd56b8dbc-njjn4\" (UID: \"580bebeb-9495-4541-8d1d-3bf6fef1a061\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" Nov 01 05:45:10.049042 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.049019 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/293a6ef2-f0c9-4390-ac82-028792a3b64d-serving-cert\") pod \"service-ca-operator-544d749c8c-vr978\" (UID: \"293a6ef2-f0c9-4390-ac82-028792a3b64d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vr978" Nov 01 05:45:10.049717 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.049700 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f9ccbc38-43f9-4396-bb59-2f746b3d639d-image-registry-private-configuration\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:10.075099 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.075065 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 05:45:10.075099 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.075083 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 05:45:10.075224 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.075164 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-vp9l2\"" Nov 01 05:45:10.075262 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.075248 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5zklw\"" Nov 01 05:45:10.088521 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.088465 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95sds" Nov 01 05:45:10.108496 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.105692 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pdtj\" (UniqueName: \"kubernetes.io/projected/293a6ef2-f0c9-4390-ac82-028792a3b64d-kube-api-access-6pdtj\") pod \"service-ca-operator-544d749c8c-vr978\" (UID: \"293a6ef2-f0c9-4390-ac82-028792a3b64d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vr978" Nov 01 05:45:10.108496 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.105724 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqqbs\" (UniqueName: \"kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-kube-api-access-vqqbs\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:10.108496 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.106180 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lmfd\" (UniqueName: \"kubernetes.io/projected/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-kube-api-access-2lmfd\") pod \"cluster-samples-operator-6ddb6c8b94-flt78\" (UID: \"320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flt78" Nov 01 05:45:10.108496 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.106697 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk8zm\" (UniqueName: \"kubernetes.io/projected/580bebeb-9495-4541-8d1d-3bf6fef1a061-kube-api-access-mk8zm\") pod \"console-operator-7cd56b8dbc-njjn4\" (UID: \"580bebeb-9495-4541-8d1d-3bf6fef1a061\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" Nov 01 05:45:10.109595 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.109137 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-bound-sa-token\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:10.109595 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.109566 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgrrf\" (UniqueName: \"kubernetes.io/projected/f4051a42-5120-4b67-b904-5db6b4897e07-kube-api-access-vgrrf\") pod \"volume-data-source-validator-58c9f5f46d-54lc7\" (UID: \"f4051a42-5120-4b67-b904-5db6b4897e07\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-54lc7" Nov 01 05:45:10.146832 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.146497 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7729d415-70e5-4be3-a9cc-bbd13e065c71-registry-certificates\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:10.146832 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.146542 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7729d415-70e5-4be3-a9cc-bbd13e065c71-installation-pull-secrets\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:10.146832 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.146578 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f98900-ff17-447f-8429-60c3e47fc7d1-tmp-dir\") pod \"dns-default-hxw5f\" (UID: \"72f98900-ff17-447f-8429-60c3e47fc7d1\") " pod="openshift-dns/dns-default-hxw5f" Nov 01 05:45:10.146832 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.146603 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hksk\" (UniqueName: \"kubernetes.io/projected/cbc9c127-21a0-4020-a093-d02b4061db5a-kube-api-access-6hksk\") pod \"insights-operator-7c4986d449-r8n4b\" (UID: \"cbc9c127-21a0-4020-a093-d02b4061db5a\") " pod="openshift-insights/insights-operator-7c4986d449-r8n4b" Nov 01 05:45:10.146832 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.146638 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzj6t\" (UniqueName: \"kubernetes.io/projected/1b19c3b0-887e-4e27-bac3-028003f1aac5-kube-api-access-fzj6t\") pod \"ingress-canary-dwklx\" (UID: \"1b19c3b0-887e-4e27-bac3-028003f1aac5\") " pod="openshift-ingress-canary/ingress-canary-dwklx" Nov 01 05:45:10.147330 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.146999 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5rfj\" (UniqueName: \"kubernetes.io/projected/72f98900-ff17-447f-8429-60c3e47fc7d1-kube-api-access-r5rfj\") pod \"dns-default-hxw5f\" (UID: \"72f98900-ff17-447f-8429-60c3e47fc7d1\") " pod="openshift-dns/dns-default-hxw5f" Nov 01 05:45:10.147330 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.147043 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbc9c127-21a0-4020-a093-d02b4061db5a-service-ca-bundle\") pod \"insights-operator-7c4986d449-r8n4b\" (UID: \"cbc9c127-21a0-4020-a093-d02b4061db5a\") " pod="openshift-insights/insights-operator-7c4986d449-r8n4b" Nov 01 05:45:10.147330 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.147080 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72qwb\" (UniqueName: \"kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-kube-api-access-72qwb\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:10.147330 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.147112 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7729d415-70e5-4be3-a9cc-bbd13e065c71-trusted-ca\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:10.147330 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.147153 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b19c3b0-887e-4e27-bac3-028003f1aac5-cert\") pod \"ingress-canary-dwklx\" (UID: \"1b19c3b0-887e-4e27-bac3-028003f1aac5\") " pod="openshift-ingress-canary/ingress-canary-dwklx" Nov 01 05:45:10.147330 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.147152 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f98900-ff17-447f-8429-60c3e47fc7d1-tmp-dir\") pod \"dns-default-hxw5f\" (UID: \"72f98900-ff17-447f-8429-60c3e47fc7d1\") " pod="openshift-dns/dns-default-hxw5f" Nov 01 05:45:10.147330 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.147183 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7729d415-70e5-4be3-a9cc-bbd13e065c71-image-registry-private-configuration\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:10.147330 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.147183 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7729d415-70e5-4be3-a9cc-bbd13e065c71-registry-certificates\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:10.147330 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.147230 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7729d415-70e5-4be3-a9cc-bbd13e065c71-ca-trust-extracted\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:10.147330 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.147261 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cbc9c127-21a0-4020-a093-d02b4061db5a-serving-cert\") pod \"insights-operator-7c4986d449-r8n4b\" (UID: \"cbc9c127-21a0-4020-a093-d02b4061db5a\") " pod="openshift-insights/insights-operator-7c4986d449-r8n4b" Nov 01 05:45:10.149537 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.147367 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbc9c127-21a0-4020-a093-d02b4061db5a-tmp\") pod \"insights-operator-7c4986d449-r8n4b\" (UID: \"cbc9c127-21a0-4020-a093-d02b4061db5a\") " pod="openshift-insights/insights-operator-7c4986d449-r8n4b" Nov 01 05:45:10.149537 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.147397 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-bound-sa-token\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:10.149537 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.147425 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-registry-tls\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:10.149537 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.147704 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbc9c127-21a0-4020-a093-d02b4061db5a-service-ca-bundle\") pod \"insights-operator-7c4986d449-r8n4b\" (UID: \"cbc9c127-21a0-4020-a093-d02b4061db5a\") " pod="openshift-insights/insights-operator-7c4986d449-r8n4b" Nov 01 05:45:10.149537 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.147854 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cbc9c127-21a0-4020-a093-d02b4061db5a-snapshots\") pod \"insights-operator-7c4986d449-r8n4b\" (UID: \"cbc9c127-21a0-4020-a093-d02b4061db5a\") " pod="openshift-insights/insights-operator-7c4986d449-r8n4b" Nov 01 05:45:10.149537 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.147903 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdc27\" (UniqueName: \"kubernetes.io/projected/f8238f6d-c793-4f76-b63b-d2d50f9a585d-kube-api-access-cdc27\") pod \"network-check-source-6f78d59f99-pgxkq\" (UID: \"f8238f6d-c793-4f76-b63b-d2d50f9a585d\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-pgxkq" Nov 01 05:45:10.149537 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.147938 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/72f98900-ff17-447f-8429-60c3e47fc7d1-config-volume\") pod \"dns-default-hxw5f\" (UID: \"72f98900-ff17-447f-8429-60c3e47fc7d1\") " pod="openshift-dns/dns-default-hxw5f" Nov 01 05:45:10.149537 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.147968 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72f98900-ff17-447f-8429-60c3e47fc7d1-metrics-tls\") pod \"dns-default-hxw5f\" (UID: \"72f98900-ff17-447f-8429-60c3e47fc7d1\") " pod="openshift-dns/dns-default-hxw5f" Nov 01 05:45:10.149537 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.147990 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbc9c127-21a0-4020-a093-d02b4061db5a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-r8n4b\" (UID: \"cbc9c127-21a0-4020-a093-d02b4061db5a\") " pod="openshift-insights/insights-operator-7c4986d449-r8n4b" Nov 01 05:45:10.149537 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.148438 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7729d415-70e5-4be3-a9cc-bbd13e065c71-trusted-ca\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:10.149537 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.148582 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbc9c127-21a0-4020-a093-d02b4061db5a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-r8n4b\" (UID: \"cbc9c127-21a0-4020-a093-d02b4061db5a\") " pod="openshift-insights/insights-operator-7c4986d449-r8n4b" Nov 01 05:45:10.149537 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:10.148965 2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 05:45:10.149537 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:10.149018 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72f98900-ff17-447f-8429-60c3e47fc7d1-metrics-tls podName:72f98900-ff17-447f-8429-60c3e47fc7d1 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:10.649001426 +0000 UTC m=+32.950627016 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/72f98900-ff17-447f-8429-60c3e47fc7d1-metrics-tls") pod "dns-default-hxw5f" (UID: "72f98900-ff17-447f-8429-60c3e47fc7d1") : secret "dns-default-metrics-tls" not found Nov 01 05:45:10.149537 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.149050 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cbc9c127-21a0-4020-a093-d02b4061db5a-snapshots\") pod \"insights-operator-7c4986d449-r8n4b\" (UID: \"cbc9c127-21a0-4020-a093-d02b4061db5a\") " pod="openshift-insights/insights-operator-7c4986d449-r8n4b" Nov 01 05:45:10.149537 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.149318 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbc9c127-21a0-4020-a093-d02b4061db5a-tmp\") pod \"insights-operator-7c4986d449-r8n4b\" (UID: \"cbc9c127-21a0-4020-a093-d02b4061db5a\") " pod="openshift-insights/insights-operator-7c4986d449-r8n4b" Nov 01 05:45:10.149537 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:10.149391 2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 05:45:10.149537 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:10.149448 2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-847d56945f-lt2tl: secret "image-registry-tls" not found Nov 01 05:45:10.150787 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:10.149550 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-registry-tls podName:7729d415-70e5-4be3-a9cc-bbd13e065c71 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:10.649539236 +0000 UTC m=+32.951164826 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-registry-tls") pod "image-registry-847d56945f-lt2tl" (UID: "7729d415-70e5-4be3-a9cc-bbd13e065c71") : secret "image-registry-tls" not found Nov 01 05:45:10.150787 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.149591 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/72f98900-ff17-447f-8429-60c3e47fc7d1-config-volume\") pod \"dns-default-hxw5f\" (UID: \"72f98900-ff17-447f-8429-60c3e47fc7d1\") " pod="openshift-dns/dns-default-hxw5f" Nov 01 05:45:10.150787 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.149958 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7729d415-70e5-4be3-a9cc-bbd13e065c71-installation-pull-secrets\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:10.156056 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.155851 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" Nov 01 05:45:10.156420 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.156373 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b19c3b0-887e-4e27-bac3-028003f1aac5-cert\") pod \"ingress-canary-dwklx\" (UID: \"1b19c3b0-887e-4e27-bac3-028003f1aac5\") " pod="openshift-ingress-canary/ingress-canary-dwklx" Nov 01 05:45:10.156994 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.156630 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cbc9c127-21a0-4020-a093-d02b4061db5a-serving-cert\") pod \"insights-operator-7c4986d449-r8n4b\" (UID: \"cbc9c127-21a0-4020-a093-d02b4061db5a\") " pod="openshift-insights/insights-operator-7c4986d449-r8n4b" Nov 01 05:45:10.157741 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.157690 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7729d415-70e5-4be3-a9cc-bbd13e065c71-ca-trust-extracted\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:10.162635 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.162589 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7729d415-70e5-4be3-a9cc-bbd13e065c71-image-registry-private-configuration\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:10.200881 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.200855 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vr978" Nov 01 05:45:10.208325 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.208274 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72qwb\" (UniqueName: \"kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-kube-api-access-72qwb\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:10.208520 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.208502 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdc27\" (UniqueName: \"kubernetes.io/projected/f8238f6d-c793-4f76-b63b-d2d50f9a585d-kube-api-access-cdc27\") pod \"network-check-source-6f78d59f99-pgxkq\" (UID: \"f8238f6d-c793-4f76-b63b-d2d50f9a585d\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-pgxkq" Nov 01 05:45:10.208722 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.208705 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzj6t\" (UniqueName: \"kubernetes.io/projected/1b19c3b0-887e-4e27-bac3-028003f1aac5-kube-api-access-fzj6t\") pod \"ingress-canary-dwklx\" (UID: \"1b19c3b0-887e-4e27-bac3-028003f1aac5\") " pod="openshift-ingress-canary/ingress-canary-dwklx" Nov 01 05:45:10.209025 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.209010 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-bound-sa-token\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:10.209539 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.209522 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5rfj\" (UniqueName: \"kubernetes.io/projected/72f98900-ff17-447f-8429-60c3e47fc7d1-kube-api-access-r5rfj\") pod \"dns-default-hxw5f\" (UID: \"72f98900-ff17-447f-8429-60c3e47fc7d1\") " pod="openshift-dns/dns-default-hxw5f" Nov 01 05:45:10.211355 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.211334 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hksk\" (UniqueName: \"kubernetes.io/projected/cbc9c127-21a0-4020-a093-d02b4061db5a-kube-api-access-6hksk\") pod \"insights-operator-7c4986d449-r8n4b\" (UID: \"cbc9c127-21a0-4020-a093-d02b4061db5a\") " pod="openshift-insights/insights-operator-7c4986d449-r8n4b" Nov 01 05:45:10.256448 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.256415 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-r8n4b" Nov 01 05:45:10.276733 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.276632 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-pgxkq" Nov 01 05:45:10.300992 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.300955 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-b2kmh" Nov 01 05:45:10.338720 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.338693 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dwklx" Nov 01 05:45:10.403049 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.402749 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-54lc7" Nov 01 05:45:10.452413 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.452380 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-service-ca-bundle\") pod \"router-default-6c5b5d5bd5-776bc\" (UID: \"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78\") " pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:10.452588 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.452425 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-metrics-certs\") pod \"router-default-6c5b5d5bd5-776bc\" (UID: \"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78\") " pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:10.452588 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.452461 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc9422c2-b041-4f6b-8495-961a9571c1e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9j8j4\" (UID: \"dc9422c2-b041-4f6b-8495-961a9571c1e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9j8j4" Nov 01 05:45:10.452722 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:10.452612 2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 05:45:10.452722 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:10.452659 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc9422c2-b041-4f6b-8495-961a9571c1e4-cluster-monitoring-operator-tls podName:dc9422c2-b041-4f6b-8495-961a9571c1e4 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:11.452646923 +0000 UTC m=+33.754272513 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dc9422c2-b041-4f6b-8495-961a9571c1e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9j8j4" (UID: "dc9422c2-b041-4f6b-8495-961a9571c1e4") : secret "cluster-monitoring-operator-tls" not found Nov 01 05:45:10.452991 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:10.452974 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-service-ca-bundle podName:fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:11.452965044 +0000 UTC m=+33.754590648 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-service-ca-bundle") pod "router-default-6c5b5d5bd5-776bc" (UID: "fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78") : configmap references non-existent config key: service-ca.crt Nov 01 05:45:10.453055 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:10.453029 2586 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 05:45:10.453109 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:10.453067 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-metrics-certs podName:fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:11.453053065 +0000 UTC m=+33.754678655 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-metrics-certs") pod "router-default-6c5b5d5bd5-776bc" (UID: "fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78") : secret "router-metrics-certs-default" not found Nov 01 05:45:10.488229 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.488198 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ls8v2"] Nov 01 05:45:10.493866 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:45:10.493833 2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31894c07_ba1e_4865_9481_58555f919bcc.slice/crio-92863f6fea65c046e369857501b96b2651633f50cf45cf0235da0464816efc46 WatchSource:0}: Error finding container 92863f6fea65c046e369857501b96b2651633f50cf45cf0235da0464816efc46: Status 404 returned error can't find the container with id 92863f6fea65c046e369857501b96b2651633f50cf45cf0235da0464816efc46 Nov 01 05:45:10.498958 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.498922 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-b2659"] Nov 01 05:45:10.499662 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.499638 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95sds"] Nov 01 05:45:10.502798 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:45:10.502776 2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod109327c0_f8f8_4dde_9cfe_489bf5a7eaad.slice/crio-cb0594fd3b22c52995ef1885c06f2ce591847c1b6c7cc9f8a992e1b203bdddf9 WatchSource:0}: Error finding container cb0594fd3b22c52995ef1885c06f2ce591847c1b6c7cc9f8a992e1b203bdddf9: Status 404 returned error can't find the container with id cb0594fd3b22c52995ef1885c06f2ce591847c1b6c7cc9f8a992e1b203bdddf9 Nov 01 05:45:10.503187 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:45:10.503168 2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e70277c_fae5_4e5a_b3d0_743d7485d2b3.slice/crio-63c68b10a5fb373166458b336ce3d061b5ac927911f63e8fee1de07f8f8ce9dc WatchSource:0}: Error finding container 63c68b10a5fb373166458b336ce3d061b5ac927911f63e8fee1de07f8f8ce9dc: Status 404 returned error can't find the container with id 63c68b10a5fb373166458b336ce3d061b5ac927911f63e8fee1de07f8f8ce9dc Nov 01 05:45:10.551229 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.551149 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-njjn4"] Nov 01 05:45:10.553984 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.553950 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/375debb4-cfe1-44ab-907b-0a34a031c2ea-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7grdp\" (UID: \"375debb4-cfe1-44ab-907b-0a34a031c2ea\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7grdp" Nov 01 05:45:10.554082 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.554036 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-flt78\" (UID: \"320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flt78" Nov 01 05:45:10.554130 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.554102 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-registry-tls\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:10.554242 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:10.554233 2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 05:45:10.554296 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:10.554245 2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f55979cbb-ngg8z: secret "image-registry-tls" not found Nov 01 05:45:10.554296 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:10.554286 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-registry-tls podName:f9ccbc38-43f9-4396-bb59-2f746b3d639d nodeName:}" failed. No retries permitted until 2026-11-01 05:45:11.554274018 +0000 UTC m=+33.855899609 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-registry-tls") pod "image-registry-5f55979cbb-ngg8z" (UID: "f9ccbc38-43f9-4396-bb59-2f746b3d639d") : secret "image-registry-tls" not found Nov 01 05:45:10.554392 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:10.554333 2586 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 05:45:10.554392 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:10.554355 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/375debb4-cfe1-44ab-907b-0a34a031c2ea-networking-console-plugin-cert podName:375debb4-cfe1-44ab-907b-0a34a031c2ea nodeName:}" failed. No retries permitted until 2026-11-01 05:45:11.55434815 +0000 UTC m=+33.855973753 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/375debb4-cfe1-44ab-907b-0a34a031c2ea-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7grdp" (UID: "375debb4-cfe1-44ab-907b-0a34a031c2ea") : secret "networking-console-plugin-cert" not found Nov 01 05:45:10.554392 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:10.554390 2586 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 05:45:10.554554 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:10.554409 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-samples-operator-tls podName:320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:11.554403962 +0000 UTC m=+33.856029565 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-flt78" (UID: "320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2") : secret "samples-operator-tls" not found Nov 01 05:45:10.593673 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.592580 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-vr978"] Nov 01 05:45:10.594761 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:45:10.594737 2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod293a6ef2_f0c9_4390_ac82_028792a3b64d.slice/crio-d9e9d026b79887b4ae830a3a92925b64430b60f6be59cfe1ea96269f13a49c92 WatchSource:0}: Error finding container d9e9d026b79887b4ae830a3a92925b64430b60f6be59cfe1ea96269f13a49c92: Status 404 returned error can't find the container with id d9e9d026b79887b4ae830a3a92925b64430b60f6be59cfe1ea96269f13a49c92 Nov 01 05:45:10.644494 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.644449 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-r8n4b"] Nov 01 05:45:10.648583 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:45:10.648541 2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc9c127_21a0_4020_a093_d02b4061db5a.slice/crio-a2520662d22589e5228afa64302b95d581c324972c73bed110953599201ea391 WatchSource:0}: Error finding container a2520662d22589e5228afa64302b95d581c324972c73bed110953599201ea391: Status 404 returned error can't find the container with id a2520662d22589e5228afa64302b95d581c324972c73bed110953599201ea391 Nov 01 05:45:10.654698 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.654664 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72f98900-ff17-447f-8429-60c3e47fc7d1-metrics-tls\") pod \"dns-default-hxw5f\" (UID: \"72f98900-ff17-447f-8429-60c3e47fc7d1\") " pod="openshift-dns/dns-default-hxw5f" Nov 01 05:45:10.654808 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:10.654797 2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 05:45:10.654888 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:10.654875 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72f98900-ff17-447f-8429-60c3e47fc7d1-metrics-tls podName:72f98900-ff17-447f-8429-60c3e47fc7d1 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:11.654854845 +0000 UTC m=+33.956480444 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/72f98900-ff17-447f-8429-60c3e47fc7d1-metrics-tls") pod "dns-default-hxw5f" (UID: "72f98900-ff17-447f-8429-60c3e47fc7d1") : secret "dns-default-metrics-tls" not found Nov 01 05:45:10.655019 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.655003 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-registry-tls\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:10.655136 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:10.655125 2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 05:45:10.655184 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:10.655137 2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-847d56945f-lt2tl: secret "image-registry-tls" not found Nov 01 05:45:10.655184 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:10.655173 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-registry-tls podName:7729d415-70e5-4be3-a9cc-bbd13e065c71 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:11.655163986 +0000 UTC m=+33.956789576 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-registry-tls") pod "image-registry-847d56945f-lt2tl" (UID: "7729d415-70e5-4be3-a9cc-bbd13e065c71") : secret "image-registry-tls" not found Nov 01 05:45:10.667679 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.667652 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-pgxkq"] Nov 01 05:45:10.670581 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:45:10.670553 2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8238f6d_c793_4f76_b63b_d2d50f9a585d.slice/crio-71a7ebb53a171eaa9b0ddeaf8dd3dc71aef2a8ee82783e0e1ef4e04e7164a4d0 WatchSource:0}: Error finding container 71a7ebb53a171eaa9b0ddeaf8dd3dc71aef2a8ee82783e0e1ef4e04e7164a4d0: Status 404 returned error can't find the container with id 71a7ebb53a171eaa9b0ddeaf8dd3dc71aef2a8ee82783e0e1ef4e04e7164a4d0 Nov 01 05:45:10.717004 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.716971 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-b2kmh"] Nov 01 05:45:10.719829 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:45:10.719795 2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cc9d6f4_202c_45bb_8a4c_4b8e936874de.slice/crio-31eaa37e8e6c221992305c756157e9d9398507aa46ab6c8bf70320aaf77c31ce WatchSource:0}: Error finding container 31eaa37e8e6c221992305c756157e9d9398507aa46ab6c8bf70320aaf77c31ce: Status 404 returned error can't find the container with id 31eaa37e8e6c221992305c756157e9d9398507aa46ab6c8bf70320aaf77c31ce Nov 01 05:45:10.748366 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.748341 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dwklx"] Nov 01 05:45:10.761508 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:45:10.758050 2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b19c3b0_887e_4e27_bac3_028003f1aac5.slice/crio-b483fa0870b72cbf1958ccb1c0d87f87e3957d4468bf8eba5fb0fa2a887831b4 WatchSource:0}: Error finding container b483fa0870b72cbf1958ccb1c0d87f87e3957d4468bf8eba5fb0fa2a887831b4: Status 404 returned error can't find the container with id b483fa0870b72cbf1958ccb1c0d87f87e3957d4468bf8eba5fb0fa2a887831b4 Nov 01 05:45:10.786804 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:10.786777 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-54lc7"] Nov 01 05:45:10.797566 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:45:10.797535 2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4051a42_5120_4b67_b904_5db6b4897e07.slice/crio-ab197e3ba9a003c80fb73bbbbbf29e0a72c041bd2197a8e41ecf584a1ea2bb59 WatchSource:0}: Error finding container ab197e3ba9a003c80fb73bbbbbf29e0a72c041bd2197a8e41ecf584a1ea2bb59: Status 404 returned error can't find the container with id ab197e3ba9a003c80fb73bbbbbf29e0a72c041bd2197a8e41ecf584a1ea2bb59 Nov 01 05:45:11.172314 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:11.172222 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:45:11.172314 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:11.172252 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:45:11.172629 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:11.172372 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljcjx" Nov 01 05:45:11.224549 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:11.224519 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 05:45:11.224549 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:11.224538 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-f4d6z\"" Nov 01 05:45:11.224957 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:11.224545 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-87mzr\"" Nov 01 05:45:11.224957 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:11.224611 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 05:45:11.224957 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:11.224661 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-v8m4f\"" Nov 01 05:45:11.374227 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:11.374192 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-pgxkq" event={"ID":"f8238f6d-c793-4f76-b63b-d2d50f9a585d","Type":"ContainerStarted","Data":"71a7ebb53a171eaa9b0ddeaf8dd3dc71aef2a8ee82783e0e1ef4e04e7164a4d0"} Nov 01 05:45:11.375610 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:11.375588 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-b2659" event={"ID":"5e70277c-fae5-4e5a-b3d0-743d7485d2b3","Type":"ContainerStarted","Data":"cc395b9dd86eba43f9b5c6f3c67a28c69c08d412ac9066fafb2b4b31b1e4e970"} Nov 01 05:45:11.375761 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:11.375633 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-b2659" event={"ID":"5e70277c-fae5-4e5a-b3d0-743d7485d2b3","Type":"ContainerStarted","Data":"63c68b10a5fb373166458b336ce3d061b5ac927911f63e8fee1de07f8f8ce9dc"} Nov 01 05:45:11.376714 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:11.376695 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-54lc7" event={"ID":"f4051a42-5120-4b67-b904-5db6b4897e07","Type":"ContainerStarted","Data":"ab197e3ba9a003c80fb73bbbbbf29e0a72c041bd2197a8e41ecf584a1ea2bb59"} Nov 01 05:45:11.377775 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:11.377758 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dwklx" event={"ID":"1b19c3b0-887e-4e27-bac3-028003f1aac5","Type":"ContainerStarted","Data":"b483fa0870b72cbf1958ccb1c0d87f87e3957d4468bf8eba5fb0fa2a887831b4"} Nov 01 05:45:11.378749 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:11.378726 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-r8n4b" event={"ID":"cbc9c127-21a0-4020-a093-d02b4061db5a","Type":"ContainerStarted","Data":"a2520662d22589e5228afa64302b95d581c324972c73bed110953599201ea391"} Nov 01 05:45:11.379746 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:11.379726 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" event={"ID":"580bebeb-9495-4541-8d1d-3bf6fef1a061","Type":"ContainerStarted","Data":"c708ffb6309a237f6a87e0ee4823bdacca0cf612f61acbbf7fc5907afbbd4c1e"} Nov 01 05:45:11.381150 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:11.381130 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ls8v2" event={"ID":"31894c07-ba1e-4865-9481-58555f919bcc","Type":"ContainerStarted","Data":"e7e2be6374d3c67b35dbbdd9f3820a30dbb3510836f7e4b91e0dc800b76cbd76"} Nov 01 05:45:11.381249 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:11.381165 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ls8v2" event={"ID":"31894c07-ba1e-4865-9481-58555f919bcc","Type":"ContainerStarted","Data":"92863f6fea65c046e369857501b96b2651633f50cf45cf0235da0464816efc46"} Nov 01 05:45:11.382524 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:11.382501 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-b2kmh" event={"ID":"1cc9d6f4-202c-45bb-8a4c-4b8e936874de","Type":"ContainerStarted","Data":"ae990b8c394456f652593fa91e7ee9507235361bacaff4dde604c77cfda1d098"} Nov 01 05:45:11.382640 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:11.382543 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-b2kmh" event={"ID":"1cc9d6f4-202c-45bb-8a4c-4b8e936874de","Type":"ContainerStarted","Data":"31eaa37e8e6c221992305c756157e9d9398507aa46ab6c8bf70320aaf77c31ce"} Nov 01 05:45:11.383588 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:11.383569 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95sds" event={"ID":"109327c0-f8f8-4dde-9cfe-489bf5a7eaad","Type":"ContainerStarted","Data":"cb0594fd3b22c52995ef1885c06f2ce591847c1b6c7cc9f8a992e1b203bdddf9"} Nov 01 05:45:11.384531 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:11.384511 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vr978" event={"ID":"293a6ef2-f0c9-4390-ac82-028792a3b64d","Type":"ContainerStarted","Data":"d9e9d026b79887b4ae830a3a92925b64430b60f6be59cfe1ea96269f13a49c92"} Nov 01 05:45:11.464312 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:11.464212 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-service-ca-bundle\") pod \"router-default-6c5b5d5bd5-776bc\" (UID: \"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78\") " pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:11.464507 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:11.464353 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-service-ca-bundle podName:fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:13.464333251 +0000 UTC m=+35.765958841 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-service-ca-bundle") pod "router-default-6c5b5d5bd5-776bc" (UID: "fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78") : configmap references non-existent config key: service-ca.crt Nov 01 05:45:11.464507 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:11.464391 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-metrics-certs\") pod \"router-default-6c5b5d5bd5-776bc\" (UID: \"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78\") " pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:11.464507 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:11.464441 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc9422c2-b041-4f6b-8495-961a9571c1e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9j8j4\" (UID: \"dc9422c2-b041-4f6b-8495-961a9571c1e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9j8j4" Nov 01 05:45:11.464670 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:11.464581 2586 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 05:45:11.464670 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:11.464646 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-metrics-certs podName:fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:13.464628574 +0000 UTC m=+35.766254163 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-metrics-certs") pod "router-default-6c5b5d5bd5-776bc" (UID: "fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78") : secret "router-metrics-certs-default" not found Nov 01 05:45:11.464670 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:11.464584 2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 05:45:11.464780 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:11.464701 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc9422c2-b041-4f6b-8495-961a9571c1e4-cluster-monitoring-operator-tls podName:dc9422c2-b041-4f6b-8495-961a9571c1e4 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:13.464691874 +0000 UTC m=+35.766317467 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dc9422c2-b041-4f6b-8495-961a9571c1e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9j8j4" (UID: "dc9422c2-b041-4f6b-8495-961a9571c1e4") : secret "cluster-monitoring-operator-tls" not found Nov 01 05:45:11.565072 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:11.565039 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/375debb4-cfe1-44ab-907b-0a34a031c2ea-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7grdp\" (UID: \"375debb4-cfe1-44ab-907b-0a34a031c2ea\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7grdp" Nov 01 05:45:11.565251 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:11.565115 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-flt78\" (UID: \"320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flt78" Nov 01 05:45:11.565251 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:11.565171 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-registry-tls\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:11.565251 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:11.565185 2586 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 05:45:11.565251 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:11.565252 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/375debb4-cfe1-44ab-907b-0a34a031c2ea-networking-console-plugin-cert podName:375debb4-cfe1-44ab-907b-0a34a031c2ea nodeName:}" failed. No retries permitted until 2026-11-01 05:45:13.565238215 +0000 UTC m=+35.866863804 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/375debb4-cfe1-44ab-907b-0a34a031c2ea-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7grdp" (UID: "375debb4-cfe1-44ab-907b-0a34a031c2ea") : secret "networking-console-plugin-cert" not found Nov 01 05:45:11.565474 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:11.565259 2586 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 05:45:11.565474 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:11.565294 2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 05:45:11.565474 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:11.565303 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-samples-operator-tls podName:320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:13.565292593 +0000 UTC m=+35.866918184 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-flt78" (UID: "320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2") : secret "samples-operator-tls" not found Nov 01 05:45:11.565474 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:11.565309 2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f55979cbb-ngg8z: secret "image-registry-tls" not found Nov 01 05:45:11.565474 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:11.565343 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-registry-tls podName:f9ccbc38-43f9-4396-bb59-2f746b3d639d nodeName:}" failed. No retries permitted until 2026-11-01 05:45:13.565334686 +0000 UTC m=+35.866960280 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-registry-tls") pod "image-registry-5f55979cbb-ngg8z" (UID: "f9ccbc38-43f9-4396-bb59-2f746b3d639d") : secret "image-registry-tls" not found Nov 01 05:45:11.666750 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:11.666301 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-registry-tls\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:11.666750 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:11.666378 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72f98900-ff17-447f-8429-60c3e47fc7d1-metrics-tls\") pod \"dns-default-hxw5f\" (UID: \"72f98900-ff17-447f-8429-60c3e47fc7d1\") " pod="openshift-dns/dns-default-hxw5f" Nov 01 05:45:11.666750 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:11.666555 2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 05:45:11.666750 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:11.666574 2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-847d56945f-lt2tl: secret "image-registry-tls" not found Nov 01 05:45:11.666750 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:11.666557 2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 05:45:11.666750 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:11.666629 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-registry-tls podName:7729d415-70e5-4be3-a9cc-bbd13e065c71 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:13.666616803 +0000 UTC m=+35.968242408 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-registry-tls") pod "image-registry-847d56945f-lt2tl" (UID: "7729d415-70e5-4be3-a9cc-bbd13e065c71") : secret "image-registry-tls" not found Nov 01 05:45:11.666750 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:11.666667 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72f98900-ff17-447f-8429-60c3e47fc7d1-metrics-tls podName:72f98900-ff17-447f-8429-60c3e47fc7d1 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:13.666651524 +0000 UTC m=+35.968277131 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/72f98900-ff17-447f-8429-60c3e47fc7d1-metrics-tls") pod "dns-default-hxw5f" (UID: "72f98900-ff17-447f-8429-60c3e47fc7d1") : secret "dns-default-metrics-tls" not found Nov 01 05:45:11.785358 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:11.785292 2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-ls8v2" podStartSLOduration=414.785274025 podStartE2EDuration="6m54.785274025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:38:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:11.784593456 +0000 UTC m=+34.086219069" watchObservedRunningTime="2026-11-01 05:45:11.785274025 +0000 UTC m=+34.086899631" Nov 01 05:45:11.901760 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:11.901619 2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-b2659" podStartSLOduration=414.901599153 podStartE2EDuration="6m54.901599153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:38:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:11.899377771 +0000 UTC m=+34.201003384" watchObservedRunningTime="2026-11-01 05:45:11.901599153 +0000 UTC m=+34.203224767" Nov 01 05:45:11.969603 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:11.969566 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7dc7486d-3188-4bb8-b907-b653caa18660-original-pull-secret\") pod \"global-pull-secret-syncer-r5l82\" (UID: \"7dc7486d-3188-4bb8-b907-b653caa18660\") " pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:45:11.969789 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:11.969622 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs\") pod \"network-metrics-daemon-tt8gq\" (UID: \"a1f5a77d-43d5-48bc-a8b5-5db8f1152853\") " pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:45:11.969789 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:11.969754 2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 01 05:45:11.969879 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:11.969812 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs podName:a1f5a77d-43d5-48bc-a8b5-5db8f1152853 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:43.969795145 +0000 UTC m=+66.271420749 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs") pod "network-metrics-daemon-tt8gq" (UID: "a1f5a77d-43d5-48bc-a8b5-5db8f1152853") : secret "metrics-daemon-secret" not found Nov 01 05:45:11.972472 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:11.972444 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7dc7486d-3188-4bb8-b907-b653caa18660-original-pull-secret\") pod \"global-pull-secret-syncer-r5l82\" (UID: \"7dc7486d-3188-4bb8-b907-b653caa18660\") " pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:45:12.070602 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:12.070281 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2xnh\" (UniqueName: \"kubernetes.io/projected/529e2d97-b167-4011-a215-02ece8519c00-kube-api-access-k2xnh\") pod \"network-check-target-ljcjx\" (UID: \"529e2d97-b167-4011-a215-02ece8519c00\") " pod="openshift-network-diagnostics/network-check-target-ljcjx" Nov 01 05:45:12.074796 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:12.074746 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2xnh\" (UniqueName: \"kubernetes.io/projected/529e2d97-b167-4011-a215-02ece8519c00-kube-api-access-k2xnh\") pod \"network-check-target-ljcjx\" (UID: \"529e2d97-b167-4011-a215-02ece8519c00\") " pod="openshift-network-diagnostics/network-check-target-ljcjx" Nov 01 05:45:12.093781 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:12.093560 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:45:12.099283 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:12.099160 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljcjx" Nov 01 05:45:12.539250 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:12.539187 2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-b2kmh" podStartSLOduration=415.53916506 podStartE2EDuration="6m55.53916506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:38:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:12.014869999 +0000 UTC m=+34.316495611" watchObservedRunningTime="2026-11-01 05:45:12.53916506 +0000 UTC m=+34.840790674" Nov 01 05:45:12.539737 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:12.539633 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r5l82"] Nov 01 05:45:12.577197 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:12.577122 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ljcjx"] Nov 01 05:45:12.580963 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:45:12.580935 2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod529e2d97_b167_4011_a215_02ece8519c00.slice/crio-5830b534fa957f41baf2438317c7ba100f3185587842dc614d50d59283bb2b03 WatchSource:0}: Error finding container 5830b534fa957f41baf2438317c7ba100f3185587842dc614d50d59283bb2b03: Status 404 returned error can't find the container with id 5830b534fa957f41baf2438317c7ba100f3185587842dc614d50d59283bb2b03 Nov 01 05:45:13.403705 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:13.403667 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ljcjx" event={"ID":"529e2d97-b167-4011-a215-02ece8519c00","Type":"ContainerStarted","Data":"5830b534fa957f41baf2438317c7ba100f3185587842dc614d50d59283bb2b03"} Nov 01 05:45:13.411440 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:13.411398 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-r5l82" event={"ID":"7dc7486d-3188-4bb8-b907-b653caa18660","Type":"ContainerStarted","Data":"4b6793ef1d065f211ddb1620589da679fd99193c116c959999b841959fce9055"} Nov 01 05:45:13.487729 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:13.487685 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-service-ca-bundle\") pod \"router-default-6c5b5d5bd5-776bc\" (UID: \"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78\") " pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:13.487889 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:13.487759 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-metrics-certs\") pod \"router-default-6c5b5d5bd5-776bc\" (UID: \"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78\") " pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:13.487889 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:13.487818 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc9422c2-b041-4f6b-8495-961a9571c1e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9j8j4\" (UID: \"dc9422c2-b041-4f6b-8495-961a9571c1e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9j8j4" Nov 01 05:45:13.488085 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:13.488043 2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 05:45:13.488190 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:13.488108 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc9422c2-b041-4f6b-8495-961a9571c1e4-cluster-monitoring-operator-tls podName:dc9422c2-b041-4f6b-8495-961a9571c1e4 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:17.488091212 +0000 UTC m=+39.789716808 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dc9422c2-b041-4f6b-8495-961a9571c1e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9j8j4" (UID: "dc9422c2-b041-4f6b-8495-961a9571c1e4") : secret "cluster-monitoring-operator-tls" not found Nov 01 05:45:13.488262 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:13.488238 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-service-ca-bundle podName:fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:17.488170862 +0000 UTC m=+39.789796452 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-service-ca-bundle") pod "router-default-6c5b5d5bd5-776bc" (UID: "fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78") : configmap references non-existent config key: service-ca.crt Nov 01 05:45:13.488339 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:13.488324 2586 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 05:45:13.488391 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:13.488371 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-metrics-certs podName:fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:17.488357962 +0000 UTC m=+39.789983554 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-metrics-certs") pod "router-default-6c5b5d5bd5-776bc" (UID: "fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78") : secret "router-metrics-certs-default" not found Nov 01 05:45:13.589346 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:13.589298 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-registry-tls\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:13.589856 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:13.589454 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/375debb4-cfe1-44ab-907b-0a34a031c2ea-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7grdp\" (UID: \"375debb4-cfe1-44ab-907b-0a34a031c2ea\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7grdp" Nov 01 05:45:13.589856 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:13.589538 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-flt78\" (UID: \"320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flt78" Nov 01 05:45:13.589856 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:13.589687 2586 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 05:45:13.589856 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:13.589753 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-samples-operator-tls podName:320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:17.589734009 +0000 UTC m=+39.891359601 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-flt78" (UID: "320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2") : secret "samples-operator-tls" not found Nov 01 05:45:13.589856 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:13.589826 2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 05:45:13.589856 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:13.589836 2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f55979cbb-ngg8z: secret "image-registry-tls" not found Nov 01 05:45:13.590189 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:13.589867 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-registry-tls podName:f9ccbc38-43f9-4396-bb59-2f746b3d639d nodeName:}" failed. No retries permitted until 2026-11-01 05:45:17.589855842 +0000 UTC m=+39.891481455 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-registry-tls") pod "image-registry-5f55979cbb-ngg8z" (UID: "f9ccbc38-43f9-4396-bb59-2f746b3d639d") : secret "image-registry-tls" not found Nov 01 05:45:13.590189 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:13.589919 2586 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 05:45:13.590189 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:13.589948 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/375debb4-cfe1-44ab-907b-0a34a031c2ea-networking-console-plugin-cert podName:375debb4-cfe1-44ab-907b-0a34a031c2ea nodeName:}" failed. No retries permitted until 2026-11-01 05:45:17.589938695 +0000 UTC m=+39.891564286 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/375debb4-cfe1-44ab-907b-0a34a031c2ea-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7grdp" (UID: "375debb4-cfe1-44ab-907b-0a34a031c2ea") : secret "networking-console-plugin-cert" not found Nov 01 05:45:13.690531 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:13.690428 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-registry-tls\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:13.690531 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:13.690523 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72f98900-ff17-447f-8429-60c3e47fc7d1-metrics-tls\") pod \"dns-default-hxw5f\" (UID: \"72f98900-ff17-447f-8429-60c3e47fc7d1\") " pod="openshift-dns/dns-default-hxw5f" Nov 01 05:45:13.690740 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:13.690679 2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 05:45:13.690740 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:13.690738 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72f98900-ff17-447f-8429-60c3e47fc7d1-metrics-tls podName:72f98900-ff17-447f-8429-60c3e47fc7d1 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:17.69072123 +0000 UTC m=+39.992346834 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/72f98900-ff17-447f-8429-60c3e47fc7d1-metrics-tls") pod "dns-default-hxw5f" (UID: "72f98900-ff17-447f-8429-60c3e47fc7d1") : secret "dns-default-metrics-tls" not found Nov 01 05:45:13.690863 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:13.690808 2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 05:45:13.690863 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:13.690819 2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-847d56945f-lt2tl: secret "image-registry-tls" not found Nov 01 05:45:13.690987 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:13.690965 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-registry-tls podName:7729d415-70e5-4be3-a9cc-bbd13e065c71 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:17.690839633 +0000 UTC m=+39.992465224 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-registry-tls") pod "image-registry-847d56945f-lt2tl" (UID: "7729d415-70e5-4be3-a9cc-bbd13e065c71") : secret "image-registry-tls" not found Nov 01 05:45:14.423118 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:14.422983 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-54lc7" event={"ID":"f4051a42-5120-4b67-b904-5db6b4897e07","Type":"ContainerStarted","Data":"466add25a8b67901abdfdc19f0fd90eb81267f1e27cb0152a7fe8f14fdc5935e"} Nov 01 05:45:14.537864 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:14.537738 2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-54lc7" podStartSLOduration=388.171708399 podStartE2EDuration="6m31.537721046s" totalImagesPullingTime="3.366012647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:38:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:14.53741803 +0000 UTC m=+36.839043644" watchObservedRunningTime="2026-11-01 05:45:14.537721046 +0000 UTC m=+36.839346661" Nov 01 05:45:15.429139 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:15.429054 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/0.log" Nov 01 05:45:15.429139 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:15.429105 2586 generic.go:356] "Generic (PLEG): container finished" podID="580bebeb-9495-4541-8d1d-3bf6fef1a061" containerID="0f3ca33a0ace7c36a1c614069f5b6d23bd9d2c5bbad483582aa340ebc86eb07b" exitCode=255 Nov 01 05:45:15.430054 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:15.430033 2586 scope.go:122] "RemoveContainer" containerID="0f3ca33a0ace7c36a1c614069f5b6d23bd9d2c5bbad483582aa340ebc86eb07b" Nov 01 05:45:15.430263 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:15.430245 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" event={"ID":"580bebeb-9495-4541-8d1d-3bf6fef1a061","Type":"ContainerDied","Data":"0f3ca33a0ace7c36a1c614069f5b6d23bd9d2c5bbad483582aa340ebc86eb07b"} Nov 01 05:45:16.439336 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:16.439248 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-r8n4b" event={"ID":"cbc9c127-21a0-4020-a093-d02b4061db5a","Type":"ContainerStarted","Data":"a9909954820f8641bc7ca4ec520502b8f4d259fb20dd5dcb8f777b08b24c49f5"} Nov 01 05:45:16.441935 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:16.441846 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/1.log" Nov 01 05:45:16.442515 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:16.442442 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/0.log" Nov 01 05:45:16.442515 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:16.442496 2586 generic.go:356] "Generic (PLEG): container finished" podID="580bebeb-9495-4541-8d1d-3bf6fef1a061" containerID="aeacbab6a94ca1ae2b5b61836714026e446cbb0d816488b4c6374f81be6410c8" exitCode=255 Nov 01 05:45:16.442681 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:16.442559 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" event={"ID":"580bebeb-9495-4541-8d1d-3bf6fef1a061","Type":"ContainerDied","Data":"aeacbab6a94ca1ae2b5b61836714026e446cbb0d816488b4c6374f81be6410c8"} Nov 01 05:45:16.442681 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:16.442612 2586 scope.go:122] "RemoveContainer" containerID="0f3ca33a0ace7c36a1c614069f5b6d23bd9d2c5bbad483582aa340ebc86eb07b" Nov 01 05:45:16.442825 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:16.442806 2586 scope.go:122] "RemoveContainer" containerID="aeacbab6a94ca1ae2b5b61836714026e446cbb0d816488b4c6374f81be6410c8" Nov 01 05:45:16.443008 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:16.442989 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-njjn4_openshift-console-operator(580bebeb-9495-4541-8d1d-3bf6fef1a061)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" podUID="580bebeb-9495-4541-8d1d-3bf6fef1a061" Nov 01 05:45:16.557142 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:16.557073 2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-r8n4b" podStartSLOduration=408.118308748 podStartE2EDuration="6m53.557052272s" totalImagesPullingTime="5.438743524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:38:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:16.555686423 +0000 UTC m=+38.857312052" watchObservedRunningTime="2026-11-01 05:45:16.557052272 +0000 UTC m=+38.858677885" Nov 01 05:45:17.370926 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.370878 2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-54tr8"] Nov 01 05:45:17.386725 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.386688 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:17.429270 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.429234 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2191a9b0-b911-46e9-8ed4-b5119e37341d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-54tr8\" (UID: \"2191a9b0-b911-46e9-8ed4-b5119e37341d\") " pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:17.429446 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.429277 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4v9k\" (UniqueName: \"kubernetes.io/projected/2191a9b0-b911-46e9-8ed4-b5119e37341d-kube-api-access-b4v9k\") pod \"insights-runtime-extractor-54tr8\" (UID: \"2191a9b0-b911-46e9-8ed4-b5119e37341d\") " pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:17.429446 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.429312 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2191a9b0-b911-46e9-8ed4-b5119e37341d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54tr8\" (UID: \"2191a9b0-b911-46e9-8ed4-b5119e37341d\") " pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:17.429446 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.429432 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2191a9b0-b911-46e9-8ed4-b5119e37341d-crio-socket\") pod \"insights-runtime-extractor-54tr8\" (UID: \"2191a9b0-b911-46e9-8ed4-b5119e37341d\") " pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:17.429584 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.429569 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2191a9b0-b911-46e9-8ed4-b5119e37341d-data-volume\") pod \"insights-runtime-extractor-54tr8\" (UID: \"2191a9b0-b911-46e9-8ed4-b5119e37341d\") " pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:17.430141 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.430114 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-54tr8"] Nov 01 05:45:17.438965 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.438907 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 05:45:17.438965 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.438925 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 05:45:17.438965 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.438962 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-74ptq\"" Nov 01 05:45:17.448412 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.448387 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/1.log" Nov 01 05:45:17.448821 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.448798 2586 scope.go:122] "RemoveContainer" containerID="aeacbab6a94ca1ae2b5b61836714026e446cbb0d816488b4c6374f81be6410c8" Nov 01 05:45:17.449029 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:17.449003 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-njjn4_openshift-console-operator(580bebeb-9495-4541-8d1d-3bf6fef1a061)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" podUID="580bebeb-9495-4541-8d1d-3bf6fef1a061" Nov 01 05:45:17.450451 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.450420 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95sds" event={"ID":"109327c0-f8f8-4dde-9cfe-489bf5a7eaad","Type":"ContainerStarted","Data":"32e2eca69bde1a4d7389195dfa332e6f9eafb097537c3b0f7f952937209a5729"} Nov 01 05:45:17.452582 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.452099 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vr978" event={"ID":"293a6ef2-f0c9-4390-ac82-028792a3b64d","Type":"ContainerStarted","Data":"f2cd7eb153d1b6552f6b136b4f34d5e2af55d9f5f2050685027b0917d10e10a3"} Nov 01 05:45:17.530772 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.530734 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-service-ca-bundle\") pod \"router-default-6c5b5d5bd5-776bc\" (UID: \"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78\") " pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:17.530937 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.530778 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2191a9b0-b911-46e9-8ed4-b5119e37341d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-54tr8\" (UID: \"2191a9b0-b911-46e9-8ed4-b5119e37341d\") " pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:17.530937 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.530802 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4v9k\" (UniqueName: \"kubernetes.io/projected/2191a9b0-b911-46e9-8ed4-b5119e37341d-kube-api-access-b4v9k\") pod \"insights-runtime-extractor-54tr8\" (UID: \"2191a9b0-b911-46e9-8ed4-b5119e37341d\") " pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:17.530937 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.530847 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2191a9b0-b911-46e9-8ed4-b5119e37341d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54tr8\" (UID: \"2191a9b0-b911-46e9-8ed4-b5119e37341d\") " pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:17.530937 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.530890 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-metrics-certs\") pod \"router-default-6c5b5d5bd5-776bc\" (UID: \"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78\") " pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:17.530937 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.530928 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2191a9b0-b911-46e9-8ed4-b5119e37341d-crio-socket\") pod \"insights-runtime-extractor-54tr8\" (UID: \"2191a9b0-b911-46e9-8ed4-b5119e37341d\") " pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:17.531176 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.530967 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc9422c2-b041-4f6b-8495-961a9571c1e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9j8j4\" (UID: \"dc9422c2-b041-4f6b-8495-961a9571c1e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9j8j4" Nov 01 05:45:17.531176 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.531133 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2191a9b0-b911-46e9-8ed4-b5119e37341d-data-volume\") pod \"insights-runtime-extractor-54tr8\" (UID: \"2191a9b0-b911-46e9-8ed4-b5119e37341d\") " pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:17.531492 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.531457 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2191a9b0-b911-46e9-8ed4-b5119e37341d-data-volume\") pod \"insights-runtime-extractor-54tr8\" (UID: \"2191a9b0-b911-46e9-8ed4-b5119e37341d\") " pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:17.532689 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.532662 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2191a9b0-b911-46e9-8ed4-b5119e37341d-crio-socket\") pod \"insights-runtime-extractor-54tr8\" (UID: \"2191a9b0-b911-46e9-8ed4-b5119e37341d\") " pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:17.532924 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:17.532778 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-service-ca-bundle podName:fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:25.532758525 +0000 UTC m=+47.834384118 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-service-ca-bundle") pod "router-default-6c5b5d5bd5-776bc" (UID: "fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78") : configmap references non-existent config key: service-ca.crt Nov 01 05:45:17.532924 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:17.532803 2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 05:45:17.532924 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:17.532834 2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 05:45:17.532924 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:17.532853 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc9422c2-b041-4f6b-8495-961a9571c1e4-cluster-monitoring-operator-tls podName:dc9422c2-b041-4f6b-8495-961a9571c1e4 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:25.532841224 +0000 UTC m=+47.834466828 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dc9422c2-b041-4f6b-8495-961a9571c1e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9j8j4" (UID: "dc9422c2-b041-4f6b-8495-961a9571c1e4") : secret "cluster-monitoring-operator-tls" not found Nov 01 05:45:17.532924 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:17.532871 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2191a9b0-b911-46e9-8ed4-b5119e37341d-kube-rbac-proxy-cm podName:2191a9b0-b911-46e9-8ed4-b5119e37341d nodeName:}" failed. No retries permitted until 2026-11-01 05:45:18.032860743 +0000 UTC m=+40.334486348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2191a9b0-b911-46e9-8ed4-b5119e37341d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-54tr8" (UID: "2191a9b0-b911-46e9-8ed4-b5119e37341d") : configmap "kube-rbac-proxy" not found Nov 01 05:45:17.533425 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:17.533407 2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 05:45:17.533510 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:17.533459 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2191a9b0-b911-46e9-8ed4-b5119e37341d-insights-runtime-extractor-tls podName:2191a9b0-b911-46e9-8ed4-b5119e37341d nodeName:}" failed. No retries permitted until 2026-11-01 05:45:18.033443581 +0000 UTC m=+40.335069189 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2191a9b0-b911-46e9-8ed4-b5119e37341d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-54tr8" (UID: "2191a9b0-b911-46e9-8ed4-b5119e37341d") : secret "insights-runtime-extractor-tls" not found Nov 01 05:45:17.533575 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:17.533536 2586 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 05:45:17.533575 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:17.533571 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-metrics-certs podName:fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:25.533558954 +0000 UTC m=+47.835184557 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-metrics-certs") pod "router-default-6c5b5d5bd5-776bc" (UID: "fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78") : secret "router-metrics-certs-default" not found Nov 01 05:45:17.594538 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.594499 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4v9k\" (UniqueName: \"kubernetes.io/projected/2191a9b0-b911-46e9-8ed4-b5119e37341d-kube-api-access-b4v9k\") pod \"insights-runtime-extractor-54tr8\" (UID: \"2191a9b0-b911-46e9-8ed4-b5119e37341d\") " pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:17.632175 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.632097 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-flt78\" (UID: \"320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flt78" Nov 01 05:45:17.632349 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.632202 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-registry-tls\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:17.632406 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.632355 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/375debb4-cfe1-44ab-907b-0a34a031c2ea-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7grdp\" (UID: \"375debb4-cfe1-44ab-907b-0a34a031c2ea\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7grdp" Nov 01 05:45:17.632564 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:17.632544 2586 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 05:45:17.632633 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:17.632603 2586 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 05:45:17.632679 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:17.632612 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/375debb4-cfe1-44ab-907b-0a34a031c2ea-networking-console-plugin-cert podName:375debb4-cfe1-44ab-907b-0a34a031c2ea nodeName:}" failed. No retries permitted until 2026-11-01 05:45:25.632594291 +0000 UTC m=+47.934219895 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/375debb4-cfe1-44ab-907b-0a34a031c2ea-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7grdp" (UID: "375debb4-cfe1-44ab-907b-0a34a031c2ea") : secret "networking-console-plugin-cert" not found Nov 01 05:45:17.632679 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:17.632677 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-samples-operator-tls podName:320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:25.632660579 +0000 UTC m=+47.934286185 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-flt78" (UID: "320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2") : secret "samples-operator-tls" not found Nov 01 05:45:17.632790 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:17.632684 2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 05:45:17.632790 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:17.632696 2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f55979cbb-ngg8z: secret "image-registry-tls" not found Nov 01 05:45:17.632790 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:17.632730 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-registry-tls podName:f9ccbc38-43f9-4396-bb59-2f746b3d639d nodeName:}" failed. No retries permitted until 2026-11-01 05:45:25.63271836 +0000 UTC m=+47.934343981 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-registry-tls") pod "image-registry-5f55979cbb-ngg8z" (UID: "f9ccbc38-43f9-4396-bb59-2f746b3d639d") : secret "image-registry-tls" not found Nov 01 05:45:17.636878 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.636797 2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vr978" podStartSLOduration=408.367429675 podStartE2EDuration="6m54.636778605s" totalImagesPullingTime="6.26934893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:38:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:17.617797708 +0000 UTC m=+39.919423320" watchObservedRunningTime="2026-11-01 05:45:17.636778605 +0000 UTC m=+39.938404218" Nov 01 05:45:17.637201 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.637167 2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-nh7q2-cs6dz"] Nov 01 05:45:17.652891 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.652860 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-nh7q2-cs6dz" Nov 01 05:45:17.700503 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.698461 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-nh7q2-cs6dz"] Nov 01 05:45:17.705514 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.705397 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-gvvtc\"" Nov 01 05:45:17.734514 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.733810 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-registry-tls\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:17.734514 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.733880 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72f98900-ff17-447f-8429-60c3e47fc7d1-metrics-tls\") pod \"dns-default-hxw5f\" (UID: \"72f98900-ff17-447f-8429-60c3e47fc7d1\") " pod="openshift-dns/dns-default-hxw5f" Nov 01 05:45:17.734514 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.733937 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-trusted-ca-bundle\") pod \"periodic-gathering-nh7q2-cs6dz\" (UID: \"47088d4b-2aa6-4f69-9847-f361e4fa5f7a\") " pod="openshift-insights/periodic-gathering-nh7q2-cs6dz" Nov 01 05:45:17.734514 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.733969 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-service-ca-bundle\") pod \"periodic-gathering-nh7q2-cs6dz\" (UID: \"47088d4b-2aa6-4f69-9847-f361e4fa5f7a\") " pod="openshift-insights/periodic-gathering-nh7q2-cs6dz" Nov 01 05:45:17.734514 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.734024 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gcvx\" (UniqueName: \"kubernetes.io/projected/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-kube-api-access-5gcvx\") pod \"periodic-gathering-nh7q2-cs6dz\" (UID: \"47088d4b-2aa6-4f69-9847-f361e4fa5f7a\") " pod="openshift-insights/periodic-gathering-nh7q2-cs6dz" Nov 01 05:45:17.734514 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.734053 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-archives-path\") pod \"periodic-gathering-nh7q2-cs6dz\" (UID: \"47088d4b-2aa6-4f69-9847-f361e4fa5f7a\") " pod="openshift-insights/periodic-gathering-nh7q2-cs6dz" Nov 01 05:45:17.734514 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:17.734305 2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 05:45:17.734514 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:17.734321 2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-847d56945f-lt2tl: secret "image-registry-tls" not found Nov 01 05:45:17.734514 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:17.734374 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-registry-tls podName:7729d415-70e5-4be3-a9cc-bbd13e065c71 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:25.734356289 +0000 UTC m=+48.035981881 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-registry-tls") pod "image-registry-847d56945f-lt2tl" (UID: "7729d415-70e5-4be3-a9cc-bbd13e065c71") : secret "image-registry-tls" not found Nov 01 05:45:17.734514 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:17.734442 2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 05:45:17.734514 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:17.734473 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72f98900-ff17-447f-8429-60c3e47fc7d1-metrics-tls podName:72f98900-ff17-447f-8429-60c3e47fc7d1 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:25.734462717 +0000 UTC m=+48.036088313 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/72f98900-ff17-447f-8429-60c3e47fc7d1-metrics-tls") pod "dns-default-hxw5f" (UID: "72f98900-ff17-447f-8429-60c3e47fc7d1") : secret "dns-default-metrics-tls" not found Nov 01 05:45:17.834989 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.834949 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gcvx\" (UniqueName: \"kubernetes.io/projected/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-kube-api-access-5gcvx\") pod \"periodic-gathering-nh7q2-cs6dz\" (UID: \"47088d4b-2aa6-4f69-9847-f361e4fa5f7a\") " pod="openshift-insights/periodic-gathering-nh7q2-cs6dz" Nov 01 05:45:17.834989 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.835000 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-archives-path\") pod \"periodic-gathering-nh7q2-cs6dz\" (UID: \"47088d4b-2aa6-4f69-9847-f361e4fa5f7a\") " pod="openshift-insights/periodic-gathering-nh7q2-cs6dz" Nov 01 05:45:17.835237 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.835177 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-trusted-ca-bundle\") pod \"periodic-gathering-nh7q2-cs6dz\" (UID: \"47088d4b-2aa6-4f69-9847-f361e4fa5f7a\") " pod="openshift-insights/periodic-gathering-nh7q2-cs6dz" Nov 01 05:45:17.835237 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.835198 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-service-ca-bundle\") pod \"periodic-gathering-nh7q2-cs6dz\" (UID: \"47088d4b-2aa6-4f69-9847-f361e4fa5f7a\") " pod="openshift-insights/periodic-gathering-nh7q2-cs6dz" Nov 01 05:45:17.835755 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.835700 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-archives-path\") pod \"periodic-gathering-nh7q2-cs6dz\" (UID: \"47088d4b-2aa6-4f69-9847-f361e4fa5f7a\") " pod="openshift-insights/periodic-gathering-nh7q2-cs6dz" Nov 01 05:45:17.835869 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.835767 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-service-ca-bundle\") pod \"periodic-gathering-nh7q2-cs6dz\" (UID: \"47088d4b-2aa6-4f69-9847-f361e4fa5f7a\") " pod="openshift-insights/periodic-gathering-nh7q2-cs6dz" Nov 01 05:45:17.836054 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.836034 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-trusted-ca-bundle\") pod \"periodic-gathering-nh7q2-cs6dz\" (UID: \"47088d4b-2aa6-4f69-9847-f361e4fa5f7a\") " pod="openshift-insights/periodic-gathering-nh7q2-cs6dz" Nov 01 05:45:17.854632 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.854576 2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95sds" podStartSLOduration=388.507389548 podStartE2EDuration="6m34.854559465s" totalImagesPullingTime="6.347169917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:38:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:17.8542691 +0000 UTC m=+40.155894712" watchObservedRunningTime="2026-11-01 05:45:17.854559465 +0000 UTC m=+40.156185077" Nov 01 05:45:17.900530 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.900499 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gcvx\" (UniqueName: \"kubernetes.io/projected/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-kube-api-access-5gcvx\") pod \"periodic-gathering-nh7q2-cs6dz\" (UID: \"47088d4b-2aa6-4f69-9847-f361e4fa5f7a\") " pod="openshift-insights/periodic-gathering-nh7q2-cs6dz" Nov 01 05:45:17.977990 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:17.977956 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-nh7q2-cs6dz" Nov 01 05:45:18.036462 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:18.036418 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2191a9b0-b911-46e9-8ed4-b5119e37341d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-54tr8\" (UID: \"2191a9b0-b911-46e9-8ed4-b5119e37341d\") " pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:18.036658 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:18.036495 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2191a9b0-b911-46e9-8ed4-b5119e37341d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54tr8\" (UID: \"2191a9b0-b911-46e9-8ed4-b5119e37341d\") " pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:18.036834 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:18.036815 2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 05:45:18.036905 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:18.036888 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2191a9b0-b911-46e9-8ed4-b5119e37341d-insights-runtime-extractor-tls podName:2191a9b0-b911-46e9-8ed4-b5119e37341d nodeName:}" failed. No retries permitted until 2026-11-01 05:45:19.036866821 +0000 UTC m=+41.338492429 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2191a9b0-b911-46e9-8ed4-b5119e37341d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-54tr8" (UID: "2191a9b0-b911-46e9-8ed4-b5119e37341d") : secret "insights-runtime-extractor-tls" not found Nov 01 05:45:18.037078 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:18.037062 2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 05:45:18.037149 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:18.037109 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2191a9b0-b911-46e9-8ed4-b5119e37341d-kube-rbac-proxy-cm podName:2191a9b0-b911-46e9-8ed4-b5119e37341d nodeName:}" failed. No retries permitted until 2026-11-01 05:45:19.037097972 +0000 UTC m=+41.338723562 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2191a9b0-b911-46e9-8ed4-b5119e37341d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-54tr8" (UID: "2191a9b0-b911-46e9-8ed4-b5119e37341d") : configmap "kube-rbac-proxy" not found Nov 01 05:45:18.369523 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:18.369469 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-nh7q2-cs6dz"] Nov 01 05:45:18.373524 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:45:18.373469 2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47088d4b_2aa6_4f69_9847_f361e4fa5f7a.slice/crio-a9dd32ace74afdc7105a4060ccacc954ea3c04bed453244c8496e0745d835782 WatchSource:0}: Error finding container a9dd32ace74afdc7105a4060ccacc954ea3c04bed453244c8496e0745d835782: Status 404 returned error can't find the container with id a9dd32ace74afdc7105a4060ccacc954ea3c04bed453244c8496e0745d835782 Nov 01 05:45:18.458837 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:18.458798 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-pgxkq" event={"ID":"f8238f6d-c793-4f76-b63b-d2d50f9a585d","Type":"ContainerStarted","Data":"4433227af0103f451d0b381bfaa2990cecc479f23e429433cfcab200012a965e"} Nov 01 05:45:18.460272 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:18.460237 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-nh7q2-cs6dz" event={"ID":"47088d4b-2aa6-4f69-9847-f361e4fa5f7a","Type":"ContainerStarted","Data":"a9dd32ace74afdc7105a4060ccacc954ea3c04bed453244c8496e0745d835782"} Nov 01 05:45:18.462962 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:18.462942 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dwklx" event={"ID":"1b19c3b0-887e-4e27-bac3-028003f1aac5","Type":"ContainerStarted","Data":"deb175dee76cc92e634827801e1c2497fa9c61389ea2d382c0a70b2e7394ea4a"} Nov 01 05:45:18.690857 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:18.690760 2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-pgxkq" podStartSLOduration=382.472994211 podStartE2EDuration="6m29.690743939s" totalImagesPullingTime="7.217749728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:38:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:18.575647687 +0000 UTC m=+40.877273303" watchObservedRunningTime="2026-11-01 05:45:18.690743939 +0000 UTC m=+40.992369547" Nov 01 05:45:18.820379 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:18.820310 2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dwklx" podStartSLOduration=3.067597027 podStartE2EDuration="9.820290818s" totalImagesPullingTime="6.752693791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:45:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:18.692498598 +0000 UTC m=+40.994124210" watchObservedRunningTime="2026-11-01 05:45:18.820290818 +0000 UTC m=+41.121916430" Nov 01 05:45:18.821150 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:18.821123 2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-wtlk8"] Nov 01 05:45:18.843588 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:18.843556 2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-n6nns"] Nov 01 05:45:18.843749 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:18.843725 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wtlk8" Nov 01 05:45:18.857074 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:18.857046 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n6nns" Nov 01 05:45:18.880988 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:18.880955 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-wtlk8"] Nov 01 05:45:18.904023 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:18.903994 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 01 05:45:18.904023 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:18.903994 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-lgfkm\"" Nov 01 05:45:18.904226 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:18.904071 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 01 05:45:18.947172 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:18.947087 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmjjx\" (UniqueName: \"kubernetes.io/projected/7db79928-e03b-4027-9cca-40191f6a300d-kube-api-access-jmjjx\") pod \"migrator-66df5d496b-wtlk8\" (UID: \"7db79928-e03b-4027-9cca-40191f6a300d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wtlk8" Nov 01 05:45:18.947172 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:18.947152 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqxsx\" (UniqueName: \"kubernetes.io/projected/baad0c0a-7a5d-454d-8ea9-4a3b27115992-kube-api-access-kqxsx\") pod \"migrator-66df5d496b-n6nns\" (UID: \"baad0c0a-7a5d-454d-8ea9-4a3b27115992\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n6nns" Nov 01 05:45:18.947679 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:18.947364 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-n6nns"] Nov 01 05:45:19.048055 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:19.048019 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqxsx\" (UniqueName: \"kubernetes.io/projected/baad0c0a-7a5d-454d-8ea9-4a3b27115992-kube-api-access-kqxsx\") pod \"migrator-66df5d496b-n6nns\" (UID: \"baad0c0a-7a5d-454d-8ea9-4a3b27115992\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n6nns" Nov 01 05:45:19.048247 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:19.048101 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2191a9b0-b911-46e9-8ed4-b5119e37341d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-54tr8\" (UID: \"2191a9b0-b911-46e9-8ed4-b5119e37341d\") " pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:19.048247 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:19.048146 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2191a9b0-b911-46e9-8ed4-b5119e37341d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54tr8\" (UID: \"2191a9b0-b911-46e9-8ed4-b5119e37341d\") " pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:19.048373 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:19.048274 2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 05:45:19.048373 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:19.048333 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2191a9b0-b911-46e9-8ed4-b5119e37341d-insights-runtime-extractor-tls podName:2191a9b0-b911-46e9-8ed4-b5119e37341d nodeName:}" failed. No retries permitted until 2026-11-01 05:45:21.048318342 +0000 UTC m=+43.349943944 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2191a9b0-b911-46e9-8ed4-b5119e37341d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-54tr8" (UID: "2191a9b0-b911-46e9-8ed4-b5119e37341d") : secret "insights-runtime-extractor-tls" not found Nov 01 05:45:19.048373 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:19.048341 2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 05:45:19.048557 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:19.048403 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2191a9b0-b911-46e9-8ed4-b5119e37341d-kube-rbac-proxy-cm podName:2191a9b0-b911-46e9-8ed4-b5119e37341d nodeName:}" failed. No retries permitted until 2026-11-01 05:45:21.048387042 +0000 UTC m=+43.350012644 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2191a9b0-b911-46e9-8ed4-b5119e37341d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-54tr8" (UID: "2191a9b0-b911-46e9-8ed4-b5119e37341d") : configmap "kube-rbac-proxy" not found Nov 01 05:45:19.048557 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:19.048348 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmjjx\" (UniqueName: \"kubernetes.io/projected/7db79928-e03b-4027-9cca-40191f6a300d-kube-api-access-jmjjx\") pod \"migrator-66df5d496b-wtlk8\" (UID: \"7db79928-e03b-4027-9cca-40191f6a300d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wtlk8" Nov 01 05:45:19.108933 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:19.108902 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqxsx\" (UniqueName: \"kubernetes.io/projected/baad0c0a-7a5d-454d-8ea9-4a3b27115992-kube-api-access-kqxsx\") pod \"migrator-66df5d496b-n6nns\" (UID: \"baad0c0a-7a5d-454d-8ea9-4a3b27115992\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n6nns" Nov 01 05:45:19.109064 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:19.108999 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmjjx\" (UniqueName: \"kubernetes.io/projected/7db79928-e03b-4027-9cca-40191f6a300d-kube-api-access-jmjjx\") pod \"migrator-66df5d496b-wtlk8\" (UID: \"7db79928-e03b-4027-9cca-40191f6a300d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wtlk8" Nov 01 05:45:19.171296 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:19.171260 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wtlk8" Nov 01 05:45:19.192370 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:19.192334 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n6nns" Nov 01 05:45:19.467958 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:19.467874 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-nh7q2-cs6dz" event={"ID":"47088d4b-2aa6-4f69-9847-f361e4fa5f7a","Type":"ContainerStarted","Data":"5cfa7ce8914b360f1d260db3a06d3c32161a7702d2da0c621a7fc7c945bc0900"} Nov 01 05:45:19.470547 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:19.470517 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ljcjx" event={"ID":"529e2d97-b167-4011-a215-02ece8519c00","Type":"ContainerStarted","Data":"accc94a8a52beff99fb61921669818f3e34ae6e52c03aa65f467b9dbc1d9bf10"} Nov 01 05:45:19.470699 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:19.470606 2586 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ljcjx" Nov 01 05:45:19.586783 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:19.586725 2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-nh7q2-cs6dz" podStartSLOduration=2.586706226 podStartE2EDuration="2.586706226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:19.584939648 +0000 UTC m=+41.886565260" watchObservedRunningTime="2026-11-01 05:45:19.586706226 +0000 UTC m=+41.888331840" Nov 01 05:45:19.672023 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:19.671996 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-wtlk8"] Nov 01 05:45:19.674843 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:45:19.674814 2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7db79928_e03b_4027_9cca_40191f6a300d.slice/crio-01d6bddcc9f3f9f4ef04ba08d2f3a3f461fa75f8f945c2ca62706c43440ec750 WatchSource:0}: Error finding container 01d6bddcc9f3f9f4ef04ba08d2f3a3f461fa75f8f945c2ca62706c43440ec750: Status 404 returned error can't find the container with id 01d6bddcc9f3f9f4ef04ba08d2f3a3f461fa75f8f945c2ca62706c43440ec750 Nov 01 05:45:19.699776 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:19.699710 2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ljcjx" podStartSLOduration=34.974542619 podStartE2EDuration="41.699693418s" totalImagesPullingTime="6.725150799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:19.699206982 +0000 UTC m=+42.000832595" watchObservedRunningTime="2026-11-01 05:45:19.699693418 +0000 UTC m=+42.001319031" Nov 01 05:45:19.723711 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:19.723642 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-n6nns"] Nov 01 05:45:19.727425 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:45:19.727395 2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaad0c0a_7a5d_454d_8ea9_4a3b27115992.slice/crio-0ac682e4e0215609e5bd32a4d45b55c4e7f9e2fe58ba5bf16998f3978c5016aa WatchSource:0}: Error finding container 0ac682e4e0215609e5bd32a4d45b55c4e7f9e2fe58ba5bf16998f3978c5016aa: Status 404 returned error can't find the container with id 0ac682e4e0215609e5bd32a4d45b55c4e7f9e2fe58ba5bf16998f3978c5016aa Nov 01 05:45:19.733828 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:19.733804 2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-zwlff"] Nov 01 05:45:19.763328 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:19.763296 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-zwlff" Nov 01 05:45:19.796897 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:19.796864 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-zwlff"] Nov 01 05:45:19.817382 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:19.816950 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 01 05:45:19.817382 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:19.817023 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 01 05:45:19.817382 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:19.817072 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 01 05:45:19.817382 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:19.817083 2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-bswc2\"" Nov 01 05:45:19.817382 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:19.817100 2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 01 05:45:19.857470 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:19.857428 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/de981773-b03b-4332-b2c1-b8d894423c57-signing-cabundle\") pod \"service-ca-5fffc97bdf-zwlff\" (UID: \"de981773-b03b-4332-b2c1-b8d894423c57\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zwlff" Nov 01 05:45:19.857652 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:19.857500 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxn5f\" (UniqueName: \"kubernetes.io/projected/de981773-b03b-4332-b2c1-b8d894423c57-kube-api-access-mxn5f\") pod \"service-ca-5fffc97bdf-zwlff\" (UID: \"de981773-b03b-4332-b2c1-b8d894423c57\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zwlff" Nov 01 05:45:19.857652 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:19.857614 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/de981773-b03b-4332-b2c1-b8d894423c57-signing-key\") pod \"service-ca-5fffc97bdf-zwlff\" (UID: \"de981773-b03b-4332-b2c1-b8d894423c57\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zwlff" Nov 01 05:45:19.959037 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:19.959006 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/de981773-b03b-4332-b2c1-b8d894423c57-signing-cabundle\") pod \"service-ca-5fffc97bdf-zwlff\" (UID: \"de981773-b03b-4332-b2c1-b8d894423c57\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zwlff" Nov 01 05:45:19.959218 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:19.959067 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxn5f\" (UniqueName: \"kubernetes.io/projected/de981773-b03b-4332-b2c1-b8d894423c57-kube-api-access-mxn5f\") pod \"service-ca-5fffc97bdf-zwlff\" (UID: \"de981773-b03b-4332-b2c1-b8d894423c57\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zwlff" Nov 01 05:45:19.959218 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:19.959151 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/de981773-b03b-4332-b2c1-b8d894423c57-signing-key\") pod \"service-ca-5fffc97bdf-zwlff\" (UID: \"de981773-b03b-4332-b2c1-b8d894423c57\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zwlff" Nov 01 05:45:19.959795 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:19.959765 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/de981773-b03b-4332-b2c1-b8d894423c57-signing-cabundle\") pod \"service-ca-5fffc97bdf-zwlff\" (UID: \"de981773-b03b-4332-b2c1-b8d894423c57\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zwlff" Nov 01 05:45:19.961901 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:19.961879 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/de981773-b03b-4332-b2c1-b8d894423c57-signing-key\") pod \"service-ca-5fffc97bdf-zwlff\" (UID: \"de981773-b03b-4332-b2c1-b8d894423c57\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zwlff" Nov 01 05:45:20.020080 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:20.020044 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxn5f\" (UniqueName: \"kubernetes.io/projected/de981773-b03b-4332-b2c1-b8d894423c57-kube-api-access-mxn5f\") pod \"service-ca-5fffc97bdf-zwlff\" (UID: \"de981773-b03b-4332-b2c1-b8d894423c57\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zwlff" Nov 01 05:45:20.089715 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:20.089677 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-zwlff" Nov 01 05:45:20.156246 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:20.156206 2586 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" Nov 01 05:45:20.156412 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:20.156301 2586 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" Nov 01 05:45:20.156773 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:20.156748 2586 scope.go:122] "RemoveContainer" containerID="aeacbab6a94ca1ae2b5b61836714026e446cbb0d816488b4c6374f81be6410c8" Nov 01 05:45:20.157012 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:20.156980 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-njjn4_openshift-console-operator(580bebeb-9495-4541-8d1d-3bf6fef1a061)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" podUID="580bebeb-9495-4541-8d1d-3bf6fef1a061" Nov 01 05:45:20.475046 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:20.474967 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n6nns" event={"ID":"baad0c0a-7a5d-454d-8ea9-4a3b27115992","Type":"ContainerStarted","Data":"0ac682e4e0215609e5bd32a4d45b55c4e7f9e2fe58ba5bf16998f3978c5016aa"} Nov 01 05:45:20.476131 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:20.476097 2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wtlk8" event={"ID":"7db79928-e03b-4027-9cca-40191f6a300d","Type":"ContainerStarted","Data":"01d6bddcc9f3f9f4ef04ba08d2f3a3f461fa75f8f945c2ca62706c43440ec750"} Nov 01 05:45:20.488968 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:20.488940 2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-zwlff"] Nov 01 05:45:20.491643 ip-10-0-143-195 kubenswrapper[2586]: W1101 05:45:20.491615 2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde981773_b03b_4332_b2c1_b8d894423c57.slice/crio-39716ebe9bcbd9040dc45c45dc40721cf58f9d698fbb5de683fe2c1b765ab803 WatchSource:0}: Error finding container 39716ebe9bcbd9040dc45c45dc40721cf58f9d698fbb5de683fe2c1b765ab803: Status 404 returned error can't find the container with id 39716ebe9bcbd9040dc45c45dc40721cf58f9d698fbb5de683fe2c1b765ab803 Nov 01 05:45:21.073142 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:21.071706 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2191a9b0-b911-46e9-8ed4-b5119e37341d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-54tr8\" (UID: \"2191a9b0-b911-46e9-8ed4-b5119e37341d\") " pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:21.073142 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:21.071762 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2191a9b0-b911-46e9-8ed4-b5119e37341d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54tr8\" (UID: \"2191a9b0-b911-46e9-8ed4-b5119e37341d\") " pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:21.073142 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:21.072062 2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 05:45:21.073142 ip-10-0-143-195 kubenswrapper[2586]: E1101 05:45:21.072129 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2191a9b0-b911-46e9-8ed4-b5119e37341d-insights-runtime-extractor-tls podName:2191a9b0-b911-46e9-8ed4-b5119e37341d nodeName:}" failed. No retries permitted until 2026-11-01 05:45:25.072107716 +0000 UTC m=+47.373733307 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2191a9b0-b911-46e9-8ed4-b5119e37341d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-54tr8" (UID: "2191a9b0-b911-46e9-8ed4-b5119e37341d") : secret "insights-runtime-extractor-tls" not found Nov 01 05:45:21.073142 ip-10-0-143-195 kubenswrapper[2586]: I1101 05:45:21.073081 2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2191a9b0-b911-46e9-8ed4-b5119e37341d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-54tr8\" (UID: \"2191a9b0-b911-46e9-8ed4-b5119e37341d\") " pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:21.238161 ip-10-0-143-195 systemd[1]: Stopping Kubernetes Kubelet... Nov 01 05:45:21.266013 ip-10-0-143-195 systemd[1]: kubelet.service: Deactivated successfully. Nov 01 05:45:21.266195 ip-10-0-143-195 systemd[1]: Stopped Kubernetes Kubelet. Nov 01 05:45:21.266248 ip-10-0-143-195 systemd[1]: kubelet.service: Consumed 3.184s CPU time. Nov 01 05:45:21.279133 ip-10-0-143-195 systemd[1]: Starting Kubernetes Kubelet... Nov 01 05:45:21.402416 ip-10-0-143-195 kubenswrapper[5040]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 05:45:21.402416 ip-10-0-143-195 kubenswrapper[5040]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 05:45:21.402416 ip-10-0-143-195 kubenswrapper[5040]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 05:45:21.402416 ip-10-0-143-195 kubenswrapper[5040]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 05:45:21.405902 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.405799 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 05:45:21.405902 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.405900 5040 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 05:45:21.405902 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.405908 5040 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 05:45:21.406061 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.405914 5040 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 05:45:21.406061 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.405919 5040 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 05:45:21.406061 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.405923 5040 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 05:45:21.406061 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.405927 5040 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 05:45:21.406061 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.405931 5040 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 05:45:21.406061 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.405935 5040 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 05:45:21.406061 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.405940 5040 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 05:45:21.406061 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.405944 5040 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 05:45:21.406061 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.405948 5040 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 05:45:21.406061 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.405952 5040 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 05:45:21.406061 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.405956 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 05:45:21.406061 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.405960 5040 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 05:45:21.406061 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.405964 5040 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 05:45:21.406061 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.405968 5040 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 05:45:21.406061 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.405973 5040 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 05:45:21.406061 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.405977 5040 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 05:45:21.406061 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.405981 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 05:45:21.406061 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.405986 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 05:45:21.406061 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.405990 5040 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 05:45:21.406061 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.405994 5040 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 05:45:21.406662 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406005 5040 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 05:45:21.406662 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406009 5040 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 05:45:21.406662 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406014 5040 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 05:45:21.406662 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406019 5040 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 05:45:21.406662 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406023 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 05:45:21.406662 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406027 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 05:45:21.406662 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406031 5040 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 05:45:21.406662 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406036 5040 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 05:45:21.406662 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406040 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 05:45:21.406662 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406044 5040 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 05:45:21.406662 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406048 5040 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 05:45:21.406662 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406053 5040 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 05:45:21.406662 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406057 5040 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 05:45:21.406662 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406062 5040 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 05:45:21.406662 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406067 5040 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 05:45:21.406662 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406072 5040 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 05:45:21.406662 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406077 5040 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 05:45:21.406662 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406081 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 05:45:21.406662 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406085 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 05:45:21.406662 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406089 5040 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 05:45:21.407207 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406095 5040 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 05:45:21.407207 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406099 5040 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 05:45:21.407207 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406103 5040 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 05:45:21.407207 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406108 5040 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 05:45:21.407207 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406113 5040 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 05:45:21.407207 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406118 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 05:45:21.407207 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406122 5040 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 05:45:21.407207 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406129 5040 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 05:45:21.407207 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406136 5040 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 05:45:21.407207 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406141 5040 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 05:45:21.407207 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406145 5040 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 05:45:21.407207 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406149 5040 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 05:45:21.407207 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406154 5040 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 05:45:21.407207 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406158 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 05:45:21.407207 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406163 5040 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 05:45:21.407207 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406167 5040 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 05:45:21.407207 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406171 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 05:45:21.407207 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406178 5040 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 05:45:21.407207 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406184 5040 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 05:45:21.407791 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406189 5040 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 05:45:21.407791 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406193 5040 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 05:45:21.407791 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406198 5040 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 05:45:21.407791 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406203 5040 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 05:45:21.407791 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406207 5040 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 05:45:21.407791 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406211 5040 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 05:45:21.407791 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406216 5040 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 05:45:21.407791 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406221 5040 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 05:45:21.407791 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406225 5040 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 05:45:21.407791 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406230 5040 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 05:45:21.407791 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406234 5040 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 05:45:21.407791 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406238 5040 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 05:45:21.407791 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406242 5040 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 05:45:21.407791 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406246 5040 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 05:45:21.407791 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406250 5040 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 05:45:21.407791 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406254 5040 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 05:45:21.407791 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406258 5040 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 05:45:21.407791 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406270 5040 feature_gate.go:386] unrecognized feature gate: Example Nov 01 05:45:21.407791 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406275 5040 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 05:45:21.407791 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406280 5040 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 05:45:21.408349 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406284 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 05:45:21.408349 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406288 5040 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 05:45:21.408349 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406292 5040 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 05:45:21.408349 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406296 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 05:45:21.408349 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406301 5040 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 05:45:21.408349 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406305 5040 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 05:45:21.408349 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406309 5040 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 05:45:21.408349 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406314 5040 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 05:45:21.408349 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406318 5040 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 05:45:21.408349 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406321 5040 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 05:45:21.408349 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406327 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 05:45:21.408349 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406332 5040 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 05:45:21.408349 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406336 5040 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 05:45:21.408349 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406340 5040 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 05:45:21.408349 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406344 5040 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 05:45:21.408349 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406348 5040 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 05:45:21.408349 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406352 5040 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 05:45:21.408349 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406357 5040 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 05:45:21.408349 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406361 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 05:45:21.408349 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406365 5040 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 05:45:21.408960 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406369 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 05:45:21.408960 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406374 5040 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 05:45:21.408960 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406379 5040 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 05:45:21.408960 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.406383 5040 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 05:45:21.408960 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407706 5040 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 05:45:21.408960 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407716 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 05:45:21.408960 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407721 5040 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 05:45:21.408960 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407725 5040 feature_gate.go:386] unrecognized feature gate: Example Nov 01 05:45:21.408960 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407729 5040 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 05:45:21.408960 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407734 5040 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 05:45:21.408960 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407745 5040 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 05:45:21.408960 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407749 5040 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 05:45:21.408960 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407753 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 05:45:21.408960 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407758 5040 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 05:45:21.408960 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407762 5040 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 05:45:21.408960 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407767 5040 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 05:45:21.408960 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407771 5040 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 05:45:21.408960 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407775 5040 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 05:45:21.408960 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407779 5040 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 05:45:21.408960 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407783 5040 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 05:45:21.409535 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407787 5040 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 05:45:21.409535 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407791 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 05:45:21.409535 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407795 5040 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 05:45:21.409535 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407801 5040 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 05:45:21.409535 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407805 5040 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 05:45:21.409535 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407809 5040 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 05:45:21.409535 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407813 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 05:45:21.409535 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407817 5040 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 05:45:21.409535 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407821 5040 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 05:45:21.409535 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407825 5040 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 05:45:21.409535 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407830 5040 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 05:45:21.409535 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407834 5040 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 05:45:21.409535 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407838 5040 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 05:45:21.409535 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407842 5040 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 05:45:21.409535 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407846 5040 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 05:45:21.409535 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407850 5040 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 05:45:21.409535 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407854 5040 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 05:45:21.409535 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407860 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 05:45:21.409535 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407864 5040 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 05:45:21.409535 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407869 5040 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 05:45:21.410171 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407874 5040 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 05:45:21.410171 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407878 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 05:45:21.410171 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407882 5040 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 05:45:21.410171 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407893 5040 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 05:45:21.410171 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407897 5040 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 05:45:21.410171 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407901 5040 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 05:45:21.410171 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407906 5040 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 05:45:21.410171 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407909 5040 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 05:45:21.410171 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407914 5040 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 05:45:21.410171 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407918 5040 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 05:45:21.410171 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407922 5040 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 05:45:21.410171 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407926 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 05:45:21.410171 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407932 5040 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 05:45:21.410171 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407936 5040 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 05:45:21.410171 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407940 5040 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 05:45:21.410171 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407944 5040 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 05:45:21.410171 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407948 5040 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 05:45:21.410171 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407953 5040 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 05:45:21.410171 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407957 5040 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 05:45:21.410171 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407960 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 05:45:21.410752 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407964 5040 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 05:45:21.410752 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407969 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 05:45:21.410752 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407973 5040 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 05:45:21.410752 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407977 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 05:45:21.410752 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407981 5040 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 05:45:21.410752 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407985 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 05:45:21.410752 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407989 5040 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 05:45:21.410752 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407993 5040 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 05:45:21.410752 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.407997 5040 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 05:45:21.410752 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408002 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 05:45:21.410752 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408006 5040 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 05:45:21.410752 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408010 5040 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 05:45:21.410752 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408014 5040 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 05:45:21.410752 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408018 5040 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 05:45:21.410752 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408025 5040 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 05:45:21.410752 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408031 5040 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 05:45:21.410752 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408036 5040 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 05:45:21.410752 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408040 5040 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 05:45:21.410752 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408044 5040 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 05:45:21.411313 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408049 5040 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 05:45:21.411313 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408053 5040 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 05:45:21.411313 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408057 5040 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 05:45:21.411313 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408061 5040 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 05:45:21.411313 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408065 5040 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 05:45:21.411313 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408070 5040 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 05:45:21.411313 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408077 5040 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 05:45:21.411313 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408082 5040 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 05:45:21.411313 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408086 5040 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 05:45:21.411313 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408090 5040 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 05:45:21.411313 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408094 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 05:45:21.411313 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408098 5040 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 05:45:21.411313 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408103 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 05:45:21.411313 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408107 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 05:45:21.411313 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408111 5040 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 05:45:21.411313 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408118 5040 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 05:45:21.411313 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408123 5040 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 05:45:21.411313 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408127 5040 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 05:45:21.411313 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408131 5040 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 05:45:21.411909 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408135 5040 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 05:45:21.411909 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408138 5040 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 05:45:21.411909 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408142 5040 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 05:45:21.411909 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408147 5040 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 05:45:21.411909 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408151 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 05:45:21.411909 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408155 5040 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 05:45:21.411909 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408159 5040 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 05:45:21.411909 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408163 5040 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 05:45:21.411909 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408167 5040 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 05:45:21.411909 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408171 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 05:45:21.411909 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408175 5040 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 05:45:21.411909 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.408179 5040 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 05:45:21.411909 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409057 5040 feature_gate.go:386] unrecognized feature gate: Example Nov 01 05:45:21.411909 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409066 5040 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 05:45:21.411909 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409070 5040 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 05:45:21.411909 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409074 5040 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 05:45:21.411909 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409079 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 05:45:21.411909 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409084 5040 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 05:45:21.411909 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409088 5040 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 05:45:21.411909 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409092 5040 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 05:45:21.412494 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409097 5040 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 05:45:21.412494 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409101 5040 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 05:45:21.412494 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409105 5040 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 05:45:21.412494 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409109 5040 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 05:45:21.412494 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409115 5040 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 05:45:21.412494 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409119 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 05:45:21.412494 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409123 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 05:45:21.412494 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409127 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 05:45:21.412494 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409132 5040 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 05:45:21.412494 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409136 5040 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 05:45:21.412494 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409140 5040 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 05:45:21.412494 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409144 5040 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 05:45:21.412494 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409149 5040 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 05:45:21.412494 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409153 5040 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 05:45:21.412494 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409157 5040 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 05:45:21.412494 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409161 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 05:45:21.412494 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409166 5040 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 05:45:21.412494 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409170 5040 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 05:45:21.412494 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409174 5040 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 05:45:21.412494 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409177 5040 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 05:45:21.413134 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409185 5040 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 05:45:21.413134 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409190 5040 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 05:45:21.413134 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409195 5040 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 05:45:21.413134 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409200 5040 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 05:45:21.413134 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409204 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 05:45:21.413134 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409208 5040 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 05:45:21.413134 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409212 5040 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 05:45:21.413134 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409216 5040 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 05:45:21.413134 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409222 5040 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 05:45:21.413134 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409226 5040 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 05:45:21.413134 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409231 5040 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 05:45:21.413134 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409235 5040 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 05:45:21.413134 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409239 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 05:45:21.413134 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409243 5040 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 05:45:21.413134 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409247 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 05:45:21.413134 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409252 5040 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 05:45:21.413134 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409256 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 05:45:21.413134 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409260 5040 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 05:45:21.413134 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409264 5040 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 05:45:21.413698 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409268 5040 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 05:45:21.413698 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409272 5040 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 05:45:21.413698 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409277 5040 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 05:45:21.413698 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409281 5040 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 05:45:21.413698 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409285 5040 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 05:45:21.413698 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409290 5040 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 05:45:21.413698 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409294 5040 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 05:45:21.413698 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409299 5040 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 05:45:21.413698 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409303 5040 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 05:45:21.413698 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409307 5040 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 05:45:21.413698 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409312 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 05:45:21.413698 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409316 5040 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 05:45:21.413698 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409320 5040 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 05:45:21.413698 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409324 5040 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 05:45:21.413698 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409329 5040 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 05:45:21.413698 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409332 5040 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 05:45:21.413698 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409341 5040 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 05:45:21.413698 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409346 5040 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 05:45:21.413698 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409350 5040 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 05:45:21.413698 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409355 5040 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 05:45:21.414272 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409359 5040 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 05:45:21.414272 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409363 5040 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 05:45:21.414272 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409367 5040 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 05:45:21.414272 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409371 5040 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 05:45:21.414272 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409375 5040 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 05:45:21.414272 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409379 5040 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 05:45:21.414272 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409383 5040 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 05:45:21.414272 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409387 5040 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 05:45:21.414272 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409391 5040 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 05:45:21.414272 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409396 5040 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 05:45:21.414272 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409400 5040 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 05:45:21.414272 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409403 5040 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 05:45:21.414272 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409408 5040 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 05:45:21.414272 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409412 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 05:45:21.414272 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409416 5040 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 05:45:21.414272 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409420 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 05:45:21.414272 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409424 5040 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 05:45:21.414272 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409428 5040 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 05:45:21.414272 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409432 5040 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 05:45:21.414272 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409437 5040 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 05:45:21.414885 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409441 5040 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 05:45:21.414885 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409445 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 05:45:21.414885 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409450 5040 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 05:45:21.414885 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409454 5040 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 05:45:21.414885 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409458 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 05:45:21.414885 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409462 5040 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 05:45:21.414885 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409466 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 05:45:21.414885 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409470 5040 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 05:45:21.414885 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409489 5040 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 05:45:21.414885 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409496 5040 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 05:45:21.414885 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409503 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 05:45:21.414885 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409508 5040 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 05:45:21.414885 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409514 5040 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 05:45:21.414885 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409519 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 05:45:21.414885 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409523 5040 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 05:45:21.414885 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409558 5040 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 05:45:21.414885 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409564 5040 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 05:45:21.414885 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409569 5040 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 05:45:21.414885 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.409574 5040 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 05:45:21.415453 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409704 5040 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 05:45:21.415453 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409720 5040 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 05:45:21.415453 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409730 5040 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 05:45:21.415453 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409737 5040 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 05:45:21.415453 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409744 5040 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 05:45:21.415453 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409749 5040 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 05:45:21.415453 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409755 5040 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 05:45:21.415453 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409762 5040 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 05:45:21.415453 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409767 5040 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 05:45:21.415453 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409772 5040 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 05:45:21.415453 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409778 5040 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 05:45:21.415453 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409783 5040 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 05:45:21.415453 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409788 5040 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 05:45:21.415453 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409793 5040 flags.go:64] FLAG: --cgroup-root="" Nov 01 05:45:21.415453 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409797 5040 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 05:45:21.415453 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409802 5040 flags.go:64] FLAG: --client-ca-file="" Nov 01 05:45:21.415453 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409807 5040 flags.go:64] FLAG: --cloud-provider="external" Nov 01 05:45:21.415453 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409814 5040 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 05:45:21.415453 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409820 5040 flags.go:64] FLAG: --cluster-domain="" Nov 01 05:45:21.415453 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409825 5040 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 05:45:21.415453 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409830 5040 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 05:45:21.415453 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409836 5040 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 05:45:21.415453 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409841 5040 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 05:45:21.415453 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409847 5040 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 05:45:21.416427 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409853 5040 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 05:45:21.416427 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409858 5040 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 05:45:21.416427 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409863 5040 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 05:45:21.416427 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409868 5040 flags.go:64] FLAG: --contention-profiling="false" Nov 01 05:45:21.416427 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409872 5040 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 05:45:21.416427 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409877 5040 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 05:45:21.416427 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409882 5040 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 05:45:21.416427 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409886 5040 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 05:45:21.416427 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409893 5040 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 05:45:21.416427 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409897 5040 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 05:45:21.416427 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409902 5040 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 05:45:21.416427 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409907 5040 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 05:45:21.416427 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409911 5040 flags.go:64] FLAG: --enable-server="true" Nov 01 05:45:21.416427 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409915 5040 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 05:45:21.416427 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409922 5040 flags.go:64] FLAG: --event-burst="100" Nov 01 05:45:21.416427 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409927 5040 flags.go:64] FLAG: --event-qps="50" Nov 01 05:45:21.416427 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409931 5040 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 05:45:21.416427 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409936 5040 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 05:45:21.416427 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409941 5040 flags.go:64] FLAG: --eviction-hard="" Nov 01 05:45:21.416427 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409948 5040 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 05:45:21.416427 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409952 5040 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 05:45:21.416427 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409957 5040 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 05:45:21.416427 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409962 5040 flags.go:64] FLAG: --eviction-soft="" Nov 01 05:45:21.416427 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409967 5040 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 05:45:21.416427 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409971 5040 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 05:45:21.417456 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409978 5040 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 05:45:21.417456 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409983 5040 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 05:45:21.417456 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409987 5040 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 05:45:21.417456 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409992 5040 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 05:45:21.417456 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.409997 5040 flags.go:64] FLAG: --feature-gates="" Nov 01 05:45:21.417456 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410004 5040 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 05:45:21.417456 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410009 5040 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 05:45:21.417456 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410015 5040 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 05:45:21.417456 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410021 5040 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 05:45:21.417456 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410026 5040 flags.go:64] FLAG: --healthz-port="10248" Nov 01 05:45:21.417456 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410031 5040 flags.go:64] FLAG: --help="false" Nov 01 05:45:21.417456 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410036 5040 flags.go:64] FLAG: --hostname-override="ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:45:21.417456 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410042 5040 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 05:45:21.417456 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410047 5040 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 05:45:21.417456 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410052 5040 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 05:45:21.417456 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410057 5040 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 05:45:21.417456 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410063 5040 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 05:45:21.417456 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410067 5040 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 05:45:21.417456 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410071 5040 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 05:45:21.417456 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410076 5040 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 05:45:21.417456 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410080 5040 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 05:45:21.417456 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410085 5040 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 05:45:21.417456 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410090 5040 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 05:45:21.418474 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410094 5040 flags.go:64] FLAG: --kube-reserved="" Nov 01 05:45:21.418474 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410099 5040 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 05:45:21.418474 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410103 5040 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 05:45:21.418474 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410109 5040 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 05:45:21.418474 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410113 5040 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 05:45:21.418474 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410117 5040 flags.go:64] FLAG: --lock-file="" Nov 01 05:45:21.418474 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410122 5040 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 05:45:21.418474 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410126 5040 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 05:45:21.418474 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410131 5040 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 05:45:21.418474 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410146 5040 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 05:45:21.418474 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410151 5040 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 05:45:21.418474 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410156 5040 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 05:45:21.418474 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410160 5040 flags.go:64] FLAG: --logging-format="text" Nov 01 05:45:21.418474 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410165 5040 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 05:45:21.418474 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410170 5040 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 05:45:21.418474 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410175 5040 flags.go:64] FLAG: --manifest-url="" Nov 01 05:45:21.418474 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410180 5040 flags.go:64] FLAG: --manifest-url-header="" Nov 01 05:45:21.418474 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410187 5040 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 05:45:21.418474 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410192 5040 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 05:45:21.418474 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410198 5040 flags.go:64] FLAG: --max-pods="110" Nov 01 05:45:21.418474 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410203 5040 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 05:45:21.418474 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410208 5040 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 05:45:21.418474 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410213 5040 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 05:45:21.418474 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410218 5040 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 05:45:21.418474 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410224 5040 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 05:45:21.419547 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410229 5040 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 05:45:21.419547 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410234 5040 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 05:45:21.419547 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410244 5040 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 05:45:21.419547 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410249 5040 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 05:45:21.419547 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410254 5040 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 05:45:21.419547 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410259 5040 flags.go:64] FLAG: --pod-cidr="" Nov 01 05:45:21.419547 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410263 5040 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 05:45:21.419547 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410267 5040 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 05:45:21.419547 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410272 5040 flags.go:64] FLAG: --pods-per-core="0" Nov 01 05:45:21.419547 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410277 5040 flags.go:64] FLAG: --port="10250" Nov 01 05:45:21.419547 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410281 5040 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 05:45:21.419547 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410286 5040 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-00708e3e63e636aaf" Nov 01 05:45:21.419547 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410291 5040 flags.go:64] FLAG: --qos-reserved="" Nov 01 05:45:21.419547 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410295 5040 flags.go:64] FLAG: --read-only-port="10255" Nov 01 05:45:21.419547 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410300 5040 flags.go:64] FLAG: --register-node="true" Nov 01 05:45:21.419547 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410305 5040 flags.go:64] FLAG: --register-with-taints="" Nov 01 05:45:21.419547 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410312 5040 flags.go:64] FLAG: --registry-burst="10" Nov 01 05:45:21.419547 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410317 5040 flags.go:64] FLAG: --registry-qps="5" Nov 01 05:45:21.419547 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410321 5040 flags.go:64] FLAG: --reserved-cpus="" Nov 01 05:45:21.419547 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410326 5040 flags.go:64] FLAG: --reserved-memory="" Nov 01 05:45:21.419547 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410332 5040 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 05:45:21.419547 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410337 5040 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 05:45:21.419547 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410341 5040 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 05:45:21.419547 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410346 5040 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 05:45:21.419547 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410351 5040 flags.go:64] FLAG: --runonce="false" Nov 01 05:45:21.419547 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410355 5040 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 05:45:21.420637 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410361 5040 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 05:45:21.420637 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410366 5040 flags.go:64] FLAG: --seccomp-default="false" Nov 01 05:45:21.420637 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410371 5040 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 05:45:21.420637 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410375 5040 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 05:45:21.420637 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410382 5040 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 05:45:21.420637 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410387 5040 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 05:45:21.420637 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410392 5040 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 05:45:21.420637 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410396 5040 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 05:45:21.420637 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410401 5040 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 05:45:21.420637 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410406 5040 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 05:45:21.420637 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410410 5040 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 05:45:21.420637 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410415 5040 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 05:45:21.420637 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410420 5040 flags.go:64] FLAG: --system-cgroups="" Nov 01 05:45:21.420637 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410425 5040 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 05:45:21.420637 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410434 5040 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 05:45:21.420637 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410440 5040 flags.go:64] FLAG: --tls-cert-file="" Nov 01 05:45:21.420637 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410444 5040 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 05:45:21.420637 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410450 5040 flags.go:64] FLAG: --tls-min-version="" Nov 01 05:45:21.420637 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410455 5040 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 05:45:21.420637 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410460 5040 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 05:45:21.420637 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410464 5040 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 05:45:21.420637 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410469 5040 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 05:45:21.420637 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410487 5040 flags.go:64] FLAG: --v="2" Nov 01 05:45:21.420637 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410495 5040 flags.go:64] FLAG: --version="false" Nov 01 05:45:21.420637 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410502 5040 flags.go:64] FLAG: --vmodule="" Nov 01 05:45:21.421704 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410508 5040 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 05:45:21.421704 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.410514 5040 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 05:45:21.421704 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410736 5040 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 05:45:21.421704 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410742 5040 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 05:45:21.421704 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410747 5040 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 05:45:21.421704 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410751 5040 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 05:45:21.421704 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410755 5040 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 05:45:21.421704 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410760 5040 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 05:45:21.421704 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410765 5040 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 05:45:21.421704 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410769 5040 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 05:45:21.421704 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410774 5040 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 05:45:21.421704 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410781 5040 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 05:45:21.421704 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410785 5040 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 05:45:21.421704 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410789 5040 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 05:45:21.421704 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410813 5040 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 05:45:21.421704 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410820 5040 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 05:45:21.421704 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410825 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 05:45:21.421704 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410829 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 05:45:21.421704 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410834 5040 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 05:45:21.421704 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410839 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 05:45:21.422537 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410843 5040 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 05:45:21.422537 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410847 5040 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 05:45:21.422537 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410851 5040 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 05:45:21.422537 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410856 5040 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 05:45:21.422537 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410860 5040 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 05:45:21.422537 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410865 5040 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 05:45:21.422537 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410869 5040 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 05:45:21.422537 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410873 5040 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 05:45:21.422537 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410877 5040 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 05:45:21.422537 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410883 5040 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 05:45:21.422537 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410887 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 05:45:21.422537 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410891 5040 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 05:45:21.422537 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410895 5040 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 05:45:21.422537 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410899 5040 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 05:45:21.422537 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410904 5040 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 05:45:21.422537 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410908 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 05:45:21.422537 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410912 5040 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 05:45:21.422537 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410917 5040 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 05:45:21.422537 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410921 5040 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 05:45:21.422537 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410925 5040 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 05:45:21.423386 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410932 5040 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 05:45:21.423386 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410939 5040 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 05:45:21.423386 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410944 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 05:45:21.423386 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410951 5040 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 05:45:21.423386 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410955 5040 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 05:45:21.423386 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410959 5040 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 05:45:21.423386 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410964 5040 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 05:45:21.423386 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410968 5040 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 05:45:21.423386 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410972 5040 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 05:45:21.423386 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410976 5040 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 05:45:21.423386 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410981 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 05:45:21.423386 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410985 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 05:45:21.423386 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410989 5040 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 05:45:21.423386 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410993 5040 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 05:45:21.423386 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.410997 5040 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 05:45:21.423386 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411001 5040 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 05:45:21.423386 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411005 5040 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 05:45:21.423386 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411011 5040 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 05:45:21.423386 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411015 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 05:45:21.424190 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411019 5040 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 05:45:21.424190 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411023 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 05:45:21.424190 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411029 5040 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 05:45:21.424190 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411034 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 05:45:21.424190 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411039 5040 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 05:45:21.424190 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411043 5040 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 05:45:21.424190 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411047 5040 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 05:45:21.424190 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411052 5040 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 05:45:21.424190 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411056 5040 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 05:45:21.424190 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411060 5040 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 05:45:21.424190 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411065 5040 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 05:45:21.424190 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411069 5040 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 05:45:21.424190 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411074 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 05:45:21.424190 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411078 5040 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 05:45:21.424190 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411082 5040 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 05:45:21.424190 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411087 5040 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 05:45:21.424190 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411093 5040 feature_gate.go:386] unrecognized feature gate: Example Nov 01 05:45:21.424190 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411097 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 05:45:21.424190 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411102 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 05:45:21.424190 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411107 5040 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 05:45:21.424985 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411112 5040 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 05:45:21.424985 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411116 5040 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 05:45:21.424985 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411120 5040 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 05:45:21.424985 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411126 5040 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 05:45:21.424985 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411133 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 05:45:21.424985 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411138 5040 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 05:45:21.424985 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411142 5040 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 05:45:21.424985 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411146 5040 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 05:45:21.424985 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411150 5040 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 05:45:21.424985 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411154 5040 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 05:45:21.424985 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411159 5040 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 05:45:21.424985 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411163 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 05:45:21.424985 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411167 5040 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 05:45:21.424985 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411171 5040 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 05:45:21.424985 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411177 5040 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 05:45:21.424985 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411182 5040 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 05:45:21.424985 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411187 5040 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 05:45:21.424985 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411191 5040 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 05:45:21.424985 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411195 5040 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 05:45:21.425781 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411199 5040 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 05:45:21.425781 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411203 5040 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 05:45:21.425781 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411207 5040 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 05:45:21.425781 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411211 5040 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 05:45:21.425781 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411216 5040 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 05:45:21.425781 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411220 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 05:45:21.425781 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411224 5040 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 05:45:21.425781 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411228 5040 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 05:45:21.425781 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411232 5040 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 05:45:21.425781 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411240 5040 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 05:45:21.425781 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.411278 5040 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 05:45:21.425781 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411332 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 05:45:21.425781 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411340 5040 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 05:45:21.425781 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411345 5040 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 05:45:21.425781 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411349 5040 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 05:45:21.425781 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411355 5040 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 05:45:21.426454 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411359 5040 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 05:45:21.426454 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411364 5040 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 05:45:21.426454 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411369 5040 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 05:45:21.426454 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411373 5040 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 05:45:21.426454 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411377 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 05:45:21.426454 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411382 5040 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 05:45:21.426454 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411386 5040 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 05:45:21.426454 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411390 5040 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 05:45:21.426454 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411394 5040 feature_gate.go:386] unrecognized feature gate: Example Nov 01 05:45:21.426454 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411399 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 05:45:21.426454 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411403 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 05:45:21.426454 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411409 5040 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 05:45:21.426454 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411414 5040 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 05:45:21.426454 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411418 5040 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 05:45:21.426454 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411422 5040 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 05:45:21.426454 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411429 5040 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 05:45:21.426454 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411435 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 05:45:21.426454 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411440 5040 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 05:45:21.426454 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411444 5040 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 05:45:21.427463 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411449 5040 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 05:45:21.427463 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411453 5040 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 05:45:21.427463 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411457 5040 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 05:45:21.427463 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411461 5040 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 05:45:21.427463 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411466 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 05:45:21.427463 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411470 5040 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 05:45:21.427463 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411474 5040 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 05:45:21.427463 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411495 5040 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 05:45:21.427463 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411500 5040 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 05:45:21.427463 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411504 5040 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 05:45:21.427463 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411508 5040 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 05:45:21.427463 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411513 5040 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 05:45:21.427463 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411517 5040 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 05:45:21.427463 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411521 5040 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 05:45:21.427463 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411525 5040 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 05:45:21.427463 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411529 5040 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 05:45:21.427463 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411534 5040 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 05:45:21.427463 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411538 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 05:45:21.427463 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411542 5040 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 05:45:21.427463 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411547 5040 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 05:45:21.428376 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411551 5040 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 05:45:21.428376 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411556 5040 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 05:45:21.428376 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411560 5040 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 05:45:21.428376 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411564 5040 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 05:45:21.428376 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411570 5040 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 05:45:21.428376 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411574 5040 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 05:45:21.428376 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411578 5040 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 05:45:21.428376 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411582 5040 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 05:45:21.428376 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411587 5040 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 05:45:21.428376 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411591 5040 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 05:45:21.428376 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411597 5040 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 05:45:21.428376 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411601 5040 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 05:45:21.428376 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411605 5040 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 05:45:21.428376 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411609 5040 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 05:45:21.428376 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411613 5040 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 05:45:21.428376 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411617 5040 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 05:45:21.428376 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411621 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 05:45:21.428376 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411625 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 05:45:21.428376 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411629 5040 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 05:45:21.428376 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411635 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 05:45:21.429227 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411639 5040 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 05:45:21.429227 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411643 5040 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 05:45:21.429227 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411648 5040 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 05:45:21.429227 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411652 5040 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 05:45:21.429227 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411656 5040 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 05:45:21.429227 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411660 5040 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 05:45:21.429227 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411664 5040 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 05:45:21.429227 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411668 5040 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 05:45:21.429227 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411672 5040 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 05:45:21.429227 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411676 5040 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 05:45:21.429227 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411680 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 05:45:21.429227 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411684 5040 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 05:45:21.429227 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411688 5040 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 05:45:21.429227 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411692 5040 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 05:45:21.429227 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411696 5040 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 05:45:21.429227 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411700 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 05:45:21.429227 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411706 5040 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 05:45:21.429227 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411710 5040 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 05:45:21.429227 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411714 5040 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 05:45:21.429227 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411718 5040 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 05:45:21.430072 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411725 5040 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 05:45:21.430072 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411730 5040 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 05:45:21.430072 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411735 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 05:45:21.430072 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411739 5040 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 05:45:21.430072 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411744 5040 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 05:45:21.430072 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411747 5040 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 05:45:21.430072 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411754 5040 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 05:45:21.430072 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411758 5040 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 05:45:21.430072 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411762 5040 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 05:45:21.430072 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411766 5040 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 05:45:21.430072 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411770 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 05:45:21.430072 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411776 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 05:45:21.430072 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411781 5040 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 05:45:21.430072 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411792 5040 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 05:45:21.430072 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411796 5040 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 05:45:21.430072 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411800 5040 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 05:45:21.430072 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411805 5040 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 05:45:21.430072 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411809 5040 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 05:45:21.430072 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411813 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 05:45:21.430891 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411817 5040 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 05:45:21.430891 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411821 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 05:45:21.430891 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.411825 5040 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 05:45:21.430891 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.417563 5040 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 05:45:21.430891 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.417581 5040 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 05:45:21.430891 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.417605 5040 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 05:45:21.430891 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.417612 5040 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 05:45:21.430891 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417696 5040 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 05:45:21.430891 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417702 5040 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 05:45:21.430891 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417707 5040 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 05:45:21.430891 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417709 5040 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 05:45:21.430891 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417713 5040 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 05:45:21.430891 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417715 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 05:45:21.430891 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417718 5040 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 05:45:21.430891 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417721 5040 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 05:45:21.430891 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417723 5040 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 05:45:21.430891 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417726 5040 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 05:45:21.430891 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417730 5040 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 05:45:21.430891 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417734 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 05:45:21.431785 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417738 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 05:45:21.431785 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417741 5040 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 05:45:21.431785 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417743 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 05:45:21.431785 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417746 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 05:45:21.431785 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417748 5040 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 05:45:21.431785 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417753 5040 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 05:45:21.431785 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417757 5040 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 05:45:21.431785 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417762 5040 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 05:45:21.431785 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417766 5040 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 05:45:21.431785 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417769 5040 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 05:45:21.431785 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417773 5040 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 05:45:21.431785 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417777 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 05:45:21.431785 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417781 5040 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 05:45:21.431785 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417786 5040 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 05:45:21.431785 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417790 5040 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 05:45:21.431785 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417794 5040 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 05:45:21.431785 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417805 5040 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 05:45:21.431785 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417810 5040 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 05:45:21.431785 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417815 5040 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 05:45:21.431785 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417820 5040 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 05:45:21.432611 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417825 5040 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 05:45:21.432611 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417830 5040 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 05:45:21.432611 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417835 5040 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 05:45:21.432611 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417840 5040 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 05:45:21.432611 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417843 5040 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 05:45:21.432611 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417845 5040 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 05:45:21.432611 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417848 5040 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 05:45:21.432611 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417850 5040 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 05:45:21.432611 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417853 5040 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 05:45:21.432611 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417856 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 05:45:21.432611 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417859 5040 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 05:45:21.432611 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417861 5040 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 05:45:21.432611 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417864 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 05:45:21.432611 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417866 5040 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 05:45:21.432611 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417869 5040 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 05:45:21.432611 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417871 5040 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 05:45:21.432611 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417874 5040 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 05:45:21.432611 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417877 5040 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 05:45:21.432611 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417880 5040 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 05:45:21.433397 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417882 5040 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 05:45:21.433397 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417885 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 05:45:21.433397 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417887 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 05:45:21.433397 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417890 5040 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 05:45:21.433397 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417893 5040 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 05:45:21.433397 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417896 5040 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 05:45:21.433397 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417899 5040 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 05:45:21.433397 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417901 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 05:45:21.433397 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417904 5040 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 05:45:21.433397 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417907 5040 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 05:45:21.433397 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417909 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 05:45:21.433397 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417911 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 05:45:21.433397 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417914 5040 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 05:45:21.433397 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417917 5040 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 05:45:21.433397 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417920 5040 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 05:45:21.433397 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417922 5040 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 05:45:21.433397 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417925 5040 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 05:45:21.433397 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417927 5040 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 05:45:21.433397 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417930 5040 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 05:45:21.433397 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417932 5040 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 05:45:21.434200 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417936 5040 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 05:45:21.434200 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417940 5040 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 05:45:21.434200 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417945 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 05:45:21.434200 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417951 5040 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 05:45:21.434200 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417956 5040 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 05:45:21.434200 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417985 5040 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 05:45:21.434200 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417989 5040 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 05:45:21.434200 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.417994 5040 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 05:45:21.434200 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418000 5040 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 05:45:21.434200 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418005 5040 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 05:45:21.434200 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418009 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 05:45:21.434200 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418013 5040 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 05:45:21.434200 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418016 5040 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 05:45:21.434200 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418019 5040 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 05:45:21.434200 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418022 5040 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 05:45:21.434200 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418024 5040 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 05:45:21.434200 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418027 5040 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 05:45:21.434200 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418029 5040 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 05:45:21.434200 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418032 5040 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 05:45:21.434991 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418034 5040 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 05:45:21.434991 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418037 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 05:45:21.434991 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418040 5040 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 05:45:21.434991 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418042 5040 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 05:45:21.434991 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418045 5040 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 05:45:21.434991 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418048 5040 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 05:45:21.434991 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418050 5040 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 05:45:21.434991 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418053 5040 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 05:45:21.434991 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418056 5040 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 05:45:21.434991 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418059 5040 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 05:45:21.434991 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418062 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 05:45:21.434991 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418064 5040 feature_gate.go:386] unrecognized feature gate: Example Nov 01 05:45:21.434991 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418067 5040 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 05:45:21.434991 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418070 5040 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 05:45:21.434991 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418072 5040 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 05:45:21.434991 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418075 5040 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 05:45:21.435673 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.418105 5040 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 05:45:21.435673 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418266 5040 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 05:45:21.435673 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418272 5040 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 05:45:21.435673 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418275 5040 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 05:45:21.435673 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418279 5040 feature_gate.go:386] unrecognized feature gate: Example Nov 01 05:45:21.435673 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418284 5040 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 05:45:21.435673 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418290 5040 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 05:45:21.435673 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418294 5040 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 05:45:21.435673 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418299 5040 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 05:45:21.435673 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418304 5040 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 05:45:21.435673 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418306 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 05:45:21.435673 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418309 5040 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 05:45:21.435673 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418312 5040 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 05:45:21.435673 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418315 5040 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 05:45:21.435673 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418317 5040 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 05:45:21.435673 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418320 5040 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 05:45:21.436356 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418322 5040 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 05:45:21.436356 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418325 5040 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 05:45:21.436356 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418329 5040 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 05:45:21.436356 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418333 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 05:45:21.436356 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418338 5040 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 05:45:21.436356 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418342 5040 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 05:45:21.436356 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418346 5040 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 05:45:21.436356 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418350 5040 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 05:45:21.436356 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418354 5040 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 05:45:21.436356 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418359 5040 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 05:45:21.436356 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418363 5040 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 05:45:21.436356 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418367 5040 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 05:45:21.436356 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418373 5040 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 05:45:21.436356 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418378 5040 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 05:45:21.436356 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418382 5040 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 05:45:21.436356 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418385 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 05:45:21.436356 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418388 5040 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 05:45:21.436356 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418391 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 05:45:21.436356 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418394 5040 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 05:45:21.436356 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418397 5040 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 05:45:21.437329 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418400 5040 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 05:45:21.437329 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418402 5040 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 05:45:21.437329 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418405 5040 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 05:45:21.437329 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418408 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 05:45:21.437329 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418411 5040 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 05:45:21.437329 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418414 5040 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 05:45:21.437329 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418416 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 05:45:21.437329 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418419 5040 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 05:45:21.437329 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418421 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 05:45:21.437329 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418424 5040 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 05:45:21.437329 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418426 5040 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 05:45:21.437329 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418429 5040 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 05:45:21.437329 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418431 5040 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 05:45:21.437329 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418436 5040 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 05:45:21.437329 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418441 5040 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 05:45:21.437329 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418447 5040 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 05:45:21.437329 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418453 5040 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 05:45:21.437329 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418458 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 05:45:21.437329 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418462 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 05:45:21.438205 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418466 5040 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 05:45:21.438205 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418469 5040 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 05:45:21.438205 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418472 5040 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 05:45:21.438205 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418475 5040 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 05:45:21.438205 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418496 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 05:45:21.438205 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418500 5040 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 05:45:21.438205 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418503 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 05:45:21.438205 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418507 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 05:45:21.438205 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418510 5040 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 05:45:21.438205 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418513 5040 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 05:45:21.438205 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418515 5040 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 05:45:21.438205 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418518 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 05:45:21.438205 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418520 5040 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 05:45:21.438205 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418523 5040 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 05:45:21.438205 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418525 5040 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 05:45:21.438205 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418528 5040 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 05:45:21.438205 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418530 5040 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 05:45:21.438205 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418534 5040 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 05:45:21.438205 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418537 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 05:45:21.438923 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418540 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 05:45:21.438923 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418542 5040 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 05:45:21.438923 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418545 5040 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 05:45:21.438923 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418547 5040 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 05:45:21.438923 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418550 5040 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 05:45:21.438923 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418552 5040 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 05:45:21.438923 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418557 5040 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 05:45:21.438923 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418561 5040 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 05:45:21.438923 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418565 5040 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 05:45:21.438923 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418569 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 05:45:21.438923 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418573 5040 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 05:45:21.438923 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418577 5040 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 05:45:21.438923 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418580 5040 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 05:45:21.438923 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418583 5040 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 05:45:21.438923 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418590 5040 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 05:45:21.438923 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418593 5040 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 05:45:21.438923 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418595 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 05:45:21.438923 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418598 5040 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 05:45:21.438923 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418601 5040 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 05:45:21.438923 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418603 5040 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 05:45:21.439411 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418606 5040 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 05:45:21.439411 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418609 5040 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 05:45:21.439411 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418611 5040 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 05:45:21.439411 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418614 5040 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 05:45:21.439411 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418617 5040 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 05:45:21.439411 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418620 5040 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 05:45:21.439411 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418623 5040 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 05:45:21.439411 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418625 5040 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 05:45:21.439411 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418628 5040 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 05:45:21.439411 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418630 5040 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 05:45:21.439411 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418633 5040 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 05:45:21.439411 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418638 5040 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 05:45:21.439411 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418643 5040 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 05:45:21.439411 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.418682 5040 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 05:45:21.439411 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418732 5040 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 05:45:21.439411 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418739 5040 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 05:45:21.439994 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418742 5040 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 05:45:21.439994 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418745 5040 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 05:45:21.439994 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418747 5040 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 05:45:21.439994 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418750 5040 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 05:45:21.439994 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418754 5040 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 05:45:21.439994 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418758 5040 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 05:45:21.439994 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418761 5040 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 05:45:21.439994 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418763 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 05:45:21.439994 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418766 5040 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 05:45:21.439994 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418768 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 05:45:21.439994 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418772 5040 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 05:45:21.439994 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418776 5040 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 05:45:21.439994 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418779 5040 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 05:45:21.439994 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418782 5040 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 05:45:21.439994 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418784 5040 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 05:45:21.439994 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418787 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 05:45:21.439994 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418789 5040 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 05:45:21.439994 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418792 5040 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 05:45:21.439994 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418794 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 05:45:21.440528 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418797 5040 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 05:45:21.440528 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418800 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 05:45:21.440528 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418802 5040 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 05:45:21.440528 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418805 5040 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 05:45:21.440528 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418809 5040 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 05:45:21.440528 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418813 5040 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 05:45:21.440528 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418821 5040 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 05:45:21.440528 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418827 5040 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 05:45:21.440528 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418833 5040 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 05:45:21.440528 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418838 5040 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 05:45:21.440528 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418841 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 05:45:21.440528 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418844 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 05:45:21.440528 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418846 5040 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 05:45:21.440528 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418849 5040 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 05:45:21.440528 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418853 5040 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 05:45:21.440528 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418857 5040 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 05:45:21.440528 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418862 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 05:45:21.440528 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418866 5040 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 05:45:21.440528 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418870 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 05:45:21.441191 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418874 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 05:45:21.441191 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418878 5040 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 05:45:21.441191 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418881 5040 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 05:45:21.441191 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418884 5040 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 05:45:21.441191 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418888 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 05:45:21.441191 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418892 5040 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 05:45:21.441191 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418895 5040 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 05:45:21.441191 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418898 5040 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 05:45:21.441191 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418900 5040 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 05:45:21.441191 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418903 5040 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 05:45:21.441191 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418905 5040 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 05:45:21.441191 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418908 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 05:45:21.441191 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418911 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 05:45:21.441191 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418914 5040 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 05:45:21.441191 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418917 5040 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 05:45:21.441191 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418919 5040 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 05:45:21.441191 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418922 5040 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 05:45:21.441191 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418924 5040 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 05:45:21.441191 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418927 5040 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 05:45:21.441191 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418930 5040 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 05:45:21.442101 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418932 5040 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 05:45:21.442101 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418935 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 05:45:21.442101 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418937 5040 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 05:45:21.442101 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418942 5040 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 05:45:21.442101 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418944 5040 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 05:45:21.442101 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418947 5040 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 05:45:21.442101 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418950 5040 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 05:45:21.442101 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418952 5040 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 05:45:21.442101 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418955 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 05:45:21.442101 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418957 5040 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 05:45:21.442101 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418960 5040 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 05:45:21.442101 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418962 5040 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 05:45:21.442101 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418965 5040 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 05:45:21.442101 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418967 5040 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 05:45:21.442101 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418970 5040 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 05:45:21.442101 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418973 5040 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 05:45:21.442101 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418977 5040 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 05:45:21.442101 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418990 5040 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 05:45:21.442101 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.418995 5040 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 05:45:21.442101 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.419001 5040 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 05:45:21.442958 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.419006 5040 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 05:45:21.442958 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.419011 5040 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 05:45:21.442958 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.419015 5040 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 05:45:21.442958 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.419020 5040 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 05:45:21.442958 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.419023 5040 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 05:45:21.442958 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.419027 5040 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 05:45:21.442958 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.419030 5040 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 05:45:21.442958 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.419033 5040 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 05:45:21.442958 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.419035 5040 feature_gate.go:386] unrecognized feature gate: Example Nov 01 05:45:21.442958 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.419038 5040 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 05:45:21.442958 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.419041 5040 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 05:45:21.442958 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.419044 5040 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 05:45:21.442958 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.419046 5040 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 05:45:21.442958 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.419049 5040 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 05:45:21.442958 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.419053 5040 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 05:45:21.442958 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.419058 5040 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 05:45:21.442958 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.419064 5040 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 05:45:21.442958 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.419068 5040 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 05:45:21.442958 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.419072 5040 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 05:45:21.442958 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.419076 5040 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 05:45:21.443967 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.419081 5040 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 05:45:21.443967 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.419086 5040 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 05:45:21.443967 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.419091 5040 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 05:45:21.443967 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.419095 5040 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 05:45:21.443967 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.419102 5040 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 05:45:21.443967 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:21.419106 5040 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 05:45:21.443967 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.419179 5040 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 05:45:21.443967 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.420336 5040 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 01 05:45:21.443967 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.420393 5040 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 01 05:45:21.443967 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.420622 5040 server.go:1014] "Starting client certificate rotation" Nov 01 05:45:21.443967 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.420725 5040 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 05:45:21.443967 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.420776 5040 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 05:39:38 +0000 UTC" deadline="2028-05-28 11:40:29.189992642 +0000 UTC" Nov 01 05:45:21.443967 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.420812 5040 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13781h55m7.769183082s" Nov 01 05:45:21.443967 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.421198 5040 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 05:45:21.443967 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.422562 5040 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 05:45:21.443967 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.424141 5040 log.go:25] "Validated CRI v1 runtime API" Nov 01 05:45:21.444668 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.425921 5040 log.go:25] "Validated CRI v1 image API" Nov 01 05:45:21.444668 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.426832 5040 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 05:45:21.444668 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.429091 5040 fs.go:135] Filesystem UUIDs: map[2f186b44-a1a3-462f-bebc-0e082a2a09a7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d516e640-1b17-4735-bd7e-1b4e7b921c04:/dev/nvme0n1p3] Nov 01 05:45:21.444832 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.429112 5040 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/01d6bddcc9f3f9f4ef04ba08d2f3a3f461fa75f8f945c2ca62706c43440ec750/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/01d6bddcc9f3f9f4ef04ba08d2f3a3f461fa75f8f945c2ca62706c43440ec750/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0ac682e4e0215609e5bd32a4d45b55c4e7f9e2fe58ba5bf16998f3978c5016aa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0ac682e4e0215609e5bd32a4d45b55c4e7f9e2fe58ba5bf16998f3978c5016aa/userdata/shm major:0 minor:347 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0f3e2e028ef2253cf0b2d40ba7a04a569991030505557c23570fea3e72cd9bb6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0f3e2e028ef2253cf0b2d40ba7a04a569991030505557c23570fea3e72cd9bb6/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/18a61660adc8698ee2d89530b878234ff5b08430b0da524a730460af2bbd2fb0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/18a61660adc8698ee2d89530b878234ff5b08430b0da524a730460af2bbd2fb0/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/31eaa37e8e6c221992305c756157e9d9398507aa46ab6c8bf70320aaf77c31ce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/31eaa37e8e6c221992305c756157e9d9398507aa46ab6c8bf70320aaf77c31ce/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/39716ebe9bcbd9040dc45c45dc40721cf58f9d698fbb5de683fe2c1b765ab803/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/39716ebe9bcbd9040dc45c45dc40721cf58f9d698fbb5de683fe2c1b765ab803/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3b51306f82e344f59688bfb3a69b3ac0b052bad3fd4178b3b55cbba30d25e728/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3b51306f82e344f59688bfb3a69b3ac0b052bad3fd4178b3b55cbba30d25e728/userdata/shm major:0 minor:75 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/40592ff5d356deee21ad5769e9533ff695bdf1eb214a9155399cf35ce47382eb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/40592ff5d356deee21ad5769e9533ff695bdf1eb214a9155399cf35ce47382eb/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b6793ef1d065f211ddb1620589da679fd99193c116c959999b841959fce9055/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b6793ef1d065f211ddb1620589da679fd99193c116c959999b841959fce9055/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5830b534fa957f41baf2438317c7ba100f3185587842dc614d50d59283bb2b03/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5830b534fa957f41baf2438317c7ba100f3185587842dc614d50d59283bb2b03/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/63c68b10a5fb373166458b336ce3d061b5ac927911f63e8fee1de07f8f8ce9dc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/63c68b10a5fb373166458b336ce3d061b5ac927911f63e8fee1de07f8f8ce9dc/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6d7d36ac248e6bfb113ab4cda9fbc9207bd58b97e154ecad53c594f39f56d265/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6d7d36ac248e6bfb113ab4cda9fbc9207bd58b97e154ecad53c594f39f56d265/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/71a7ebb53a171eaa9b0ddeaf8dd3dc71aef2a8ee82783e0e1ef4e04e7164a4d0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/71a7ebb53a171eaa9b0ddeaf8dd3dc71aef2a8ee82783e0e1ef4e04e7164a4d0/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/746049f9b24a68fedc050b50f4332cafda0a4d614d8feaa7cc941e321dcf62c9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/746049f9b24a68fedc050b50f4332cafda0a4d614d8feaa7cc941e321dcf62c9/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/92863f6fea65c046e369857501b96b2651633f50cf45cf0235da0464816efc46/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/92863f6fea65c046e369857501b96b2651633f50cf45cf0235da0464816efc46/userdata/shm major:0 minor:212 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/944070a7add267d0399adb6828fa1d1240081b78d60309c0db13f195a30023b9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/944070a7add267d0399adb6828fa1d1240081b78d60309c0db13f195a30023b9/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9fd52c62387ec79eddc2fc1c21b7174cd35bdb356617695d27af2b5a992b9442/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9fd52c62387ec79eddc2fc1c21b7174cd35bdb356617695d27af2b5a992b9442/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a2520662d22589e5228afa64302b95d581c324972c73bed110953599201ea391/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a2520662d22589e5228afa64302b95d581c324972c73bed110953599201ea391/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a3c54009ff4e69efff07e45faa1e917b0aec9d2b2f3fd61935987368b12409c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a3c54009ff4e69efff07e45faa1e917b0aec9d2b2f3fd61935987368b12409c5/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a9dd32ace74afdc7105a4060ccacc954ea3c04bed453244c8496e0745d835782/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a9dd32ace74afdc7105a4060ccacc954ea3c04bed453244c8496e0745d835782/userdata/shm major:0 minor:335 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ab197e3ba9a003c80fb73bbbbbf29e0a72c041bd2197a8e41ecf584a1ea2bb59/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ab197e3ba9a003c80fb73bbbbbf29e0a72c041bd2197a8e41ecf584a1ea2bb59/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b483fa0870b72cbf1958ccb1c0d87f87e3957d4468bf8eba5fb0fa2a887831b4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b483fa0870b72cbf1958ccb1c0d87f87e3957d4468bf8eba5fb0fa2a887831b4/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf8defe53821b5fe41a5e5da1523fee6fa713a3253535d5f90f74b6afe37c04c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf8defe53821b5fe41a5e5da1523fee6fa713a3253535d5f90f74b6afe37c04c/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c708ffb6309a237f6a87e0ee4823bdacca0cf612f61acbbf7fc5907afbbd4c1e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c708ffb6309a237f6a87e0ee4823bdacca0cf612f61acbbf7fc5907afbbd4c1e/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cb0594fd3b22c52995ef1885c06f2ce591847c1b6c7cc9f8a992e1b203bdddf9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cb0594fd3b22c52995ef1885c06f2ce591847c1b6c7cc9f8a992e1b203bdddf9/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d9e9d026b79887b4ae830a3a92925b64430b60f6be59cfe1ea96269f13a49c92/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d9e9d026b79887b4ae830a3a92925b64430b60f6be59cfe1ea96269f13a49c92/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0719dc90-dbae-425a-8439-812425e1b54c/volumes/kubernetes.io~projected/kube-api-access-mdrd9:{mountpoint:/var/lib/kubelet/pods/0719dc90-dbae-425a-8439-812425e1b54c/volumes/kubernetes.io~projected/kube-api-access-mdrd9 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/109327c0-f8f8-4dde-9cfe-489bf5a7eaad/volumes/kubernetes.io~projected/kube-api-access-j64kh:{mountpoint:/var/lib/kubelet/pods/109327c0-f8f8-4dde-9cfe-489bf5a7eaad/volumes/kubernetes.io~projected/kube-api-access-j64kh major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/109327c0-f8f8-4dde-9cfe-489bf5a7eaad/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/109327c0-f8f8-4dde-9cfe-489bf5a7eaad/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/197fb18c-6703-4deb-93b3-0186d7ceed67/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/197fb18c-6703-4deb-93b3-0186d7ceed67/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b19c3b0-887e-4e27-bac3-028003f1aac5/volumes/kubernetes.io~projected/kube-api-access-fzj6t:{mountpoint:/var/lib/kubelet/pods/1b19c3b0-887e-4e27-bac3-028003f1aac5/volumes/kubernetes.io~projected/kube-api-access-fzj6t major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b19c3b0-887e-4e27-bac3-028003f1aac5/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/1b19c3b0-887e-4e27-bac3-028003f1aac5/volumes/kubernetes.io~secret/cert major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1cc9d6f4-202c-45bb-8a4c-4b8e936874de/volumes/kubernetes.io~projected/kube-api-access-n2nbg:{mountpoint:/var/lib/kubelet/pods/1cc9d6f4-202c-45bb-8a4c-4b8e936874de/volumes/kubernetes.io~projected/kube-api-access-n2nbg major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21519345-86b8-441e-a395-1e8385cd3732/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/21519345-86b8-441e-a395-1e8385cd3732/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21519345-86b8-441e-a395-1e8385cd3732/volumes/kubernetes.io~projected/kube-api-access-g6xv7:{mountpoint:/var/lib/kubelet/pods/21519345-86b8-441e-a395-1e8385cd3732/volumes/kubernetes.io~projected/kube-api-access-g6xv7 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21519345-86b8-441e-a395-1e8385cd3732/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/21519345-86b8-441e-a395-1e8385cd3732/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2191a9b0-b911-46e9-8ed4-b5119e37341d/volumes/kubernetes.io~projected/kube-api-access-b4v9k:{mountpoint:/var/lib/kubelet/pods/2191a9b0-b911-46e9-8ed4-b5119e37341d/volumes/kubernetes.io~projected/kube-api-access-b4v9k major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/293a6ef2-f0c9-4390-ac82-028792a3b64d/volumes/kubernetes.io~projected/kube-api-access-6pdtj:{mountpoint:/var/lib/kubelet/pods/293a6ef2-f0c9-4390-ac82-028792a3b64d/volumes/kubernetes.io~projected/kube-api-access-6pdtj major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/293a6ef2-f0c9-4390-ac82-028792a3b64d/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/293a6ef2-f0c9-4390-ac82-028792a3b64d/volumes/kubernetes.io~secret/serving-cert major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2acbc647-2361-4935-b503-5bd9af74f822/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/2acbc647-2361-4935-b503-5bd9af74f822/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2acbc647-2361-4935-b503-5bd9af74f822/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/2acbc647-2361-4935-b503-5bd9af74f822/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2acbc647-2361-4935-b503-5bd9af74f822/volumes/kubernetes.io~projected/kube-api-access-4s56k:{mountpoint:/var/lib/kubelet/pods/2acbc647-2361-4935-b503-5bd9af74f822/volumes/kubernetes.io~projected/kube-api-access-4s56k major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31894c07-ba1e-4865-9481-58555f919bcc/volumes/kubernetes.io~projected/kube-api-access-t6r8g:{mountpoint:/var/lib/kubelet/pods/31894c07-ba1e-4865-9481-58555f919bcc/volumes/kubernetes.io~projected/kube-api-access-t6r8g major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2/volumes/kubernetes.io~projected/kube-api-access-2lmfd:{mountpoint:/var/lib/kubelet/pods/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2/volumes/kubernetes.io~projected/kube-api-access-2lmfd major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47088d4b-2aa6-4f69-9847-f361e4fa5f7a/volumes/kubernetes.io~projected/kube-api-access-5gcvx:{mountpoint:/var/lib/kubelet/pods/47088d4b-2aa6-4f69-9847-f361e4fa5f7a/volumes/kubernetes.io~projected/kube-api-access-5gcvx major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/529e2d97-b167-4011-a215-02ece8519c00/volumes/kubernetes.io~projected/kube-api-access-k2xnh:{mountpoint:/var/lib/kubelet/pods/529e2d97-b167-4011-a215-02ece8519c00/volumes/kubernetes.io~projected/kube-api-access-k2xnh major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/580bebeb-9495-4541-8d1d-3bf6fef1a061/volumes/kubernetes.io~projected/kube-api-access-mk8zm:{mountpoint:/var/lib/kubelet/pods/580bebeb-9495-4541-8d1d-3bf6fef1a061/volumes/kubernetes.io~projected/kube-api-access-mk8zm major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/580bebeb-9495-4541-8d1d-3bf6fef1a061/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/580bebeb-9495-4541-8d1d-3bf6fef1a061/volumes/kubernetes.io~secret/serving-cert major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e70277c-fae5-4e5a-b3d0-743d7485d2b3/volumes/kubernetes.io~projected/kube-api-access-swfdf:{mountpoint:/var/lib/kubelet/pods/5e70277c-fae5-4e5a-b3d0-743d7485d2b3/volumes/kubernetes.io~projected/kube-api-access-swfdf major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67ae987f-79b9-4852-98bd-e0537c412b17/volumes/kubernetes.io~projected/kube-api-access-7gbgq:{mountpoint:/var/lib/kubelet/pods/67ae987f-79b9-4852-98bd-e0537c412b17/volumes/kubernetes.io~projected/kube-api-access-7gbgq major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6cd67a78-09f1-4988-99b0-1ac9097f2f39/volumes/kubernetes.io~projected/kube-api-access-kz6tw:{mountpoint:/var/lib/kubelet/pods/6cd67a78-09f1-4988-99b0-1ac9097f2f39/volumes/kubernetes.io~projected/kube-api-access-kz6tw major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72f98900-ff17-447f-8429-60c3e47fc7d1/volumes/kubernetes.io~projected/kube-api-access-r5rfj:{mountpoint:/var/lib/kubelet/pods/72f98900-ff17-447f-8429-60c3e47fc7d1/volumes/kubernetes.io~projected/kube-api-access-r5rfj major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7729d415-70e5-4be3-a9cc-bbd13e065c71/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/7729d415-70e5-4be3-a9cc-bbd13e065c71/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7729d415-70e5-4be3-a9cc-bbd13e065c71/volumes/kubernetes.io~projected/kube-api-access-72qwb:{mountpoint:/var/lib/kubelet/pods/7729d415-70e5-4be3-a9cc-bbd13e065c71/volumes/kubernetes.io~projected/kube-api-access-72qwb major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7729d415-70e5-4be3-a9cc-bbd13e065c71/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/7729d415-70e5-4be3-a9cc-bbd13e065c71/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7729d415-70e5-4be3-a9cc-bbd13e065c71/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/7729d415-70e5-4be3-a9cc-bbd13e065c71/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7db79928-e03b-4027-9cca-40191f6a300d/volumes/kubernetes.io~projected/kube-api-access-jmjjx:{mountpoint:/var/lib/kubelet/pods/7db79928-e03b-4027-9cca-40191f6a300d/volumes/kubernetes.io~projected/kube-api-access-jmjjx major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7dc7486d-3188-4bb8-b907-b653caa18660/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/7dc7486d-3188-4bb8-b907-b653caa18660/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1f5a77d-43d5-48bc-a8b5-5db8f1152853/volumes/kubernetes.io~projected/kube-api-access-f8fz2:{mountpoint:/var/lib/kubelet/pods/a1f5a77d-43d5-48bc-a8b5-5db8f1152853/volumes/kubernetes.io~projected/kube-api-access-f8fz2 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/baad0c0a-7a5d-454d-8ea9-4a3b27115992/volumes/kubernetes.io~projected/kube-api-access-kqxsx:{mountpoint:/var/lib/kubelet/pods/baad0c0a-7a5d-454d-8ea9-4a3b27115992/volumes/kubernetes.io~projected/kube-api-access-kqxsx major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbc9c127-21a0-4020-a093-d02b4061db5a/volumes/kubernetes.io~projected/kube-api-access-6hksk:{mountpoint:/var/lib/kubelet/pods/cbc9c127-21a0-4020-a093-d02b4061db5a/volumes/kubernetes.io~projected/kube-api-access-6hksk major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbc9c127-21a0-4020-a093-d02b4061db5a/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/cbc9c127-21a0-4020-a093-d02b4061db5a/volumes/kubernetes.io~secret/serving-cert major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cde21439-a494-47cd-8cc5-161fc96a6701/volumes/kubernetes.io~projected/kube-api-access-8dbh2:{mountpoint:/var/lib/kubelet/pods/cde21439-a494-47cd-8cc5-161fc96a6701/volumes/kubernetes.io~projected/kube-api-access-8dbh2 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc9422c2-b041-4f6b-8495-961a9571c1e4/volumes/kubernetes.io~projected/kube-api-access-jsp9k:{mountpoint:/var/lib/kubelet/pods/dc9422c2-b041-4f6b-8495-961a9571c1e4/volumes/kubernetes.io~projected/kube-api-access-jsp9k major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de981773-b03b-4332-b2c1-b8d894423c57/volumes/kubernetes.io~projected/kube-api-access-mxn5f:{mountpoint:/var/lib/kubelet/pods/de981773-b03b-4332-b2c1-b8d894423c57/volumes/kubernetes.io~projected/kube-api-access-mxn5f major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de981773-b03b-4332-b2c1-b8d894423c57/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/de981773-b03b-4332-b2c1-b8d894423c57/volumes/kubernetes.io~secret/signing-key major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8/volumes/kubernetes.io~projected/kube-api-access-rrtw2:{mountpoint:/var/lib/kubelet/pods/ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8/volumes/kubernetes.io~projected/kube-api-access-rrtw2 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4051a42-5120-4b67-b904-5db6b4897e07/volumes/kubernetes.io~projected/kube-api-access-vgrrf:{mountpoint:/var/lib/kubelet/pods/f4051a42-5120-4b67-b904-5db6b4897e07/volumes/kubernetes.io~projected/kube-api-access-vgrrf major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f5705a40-d3e4-40a8-a561-17c435f388b1/volumes/kubernetes.io~projected/kube-api-access-dzxml:{mountpoint:/var/lib/kubelet/pods/f5705a40-d3e4-40a8-a561-17c435f388b1/volumes/kubernetes.io~projected/kube-api-access-dzxml major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8238f6d-c793-4f76-b63b-d2d50f9a585d/volumes/kubernetes.io~projected/kube-api-access-cdc27:{mountpoint:/var/lib/kubelet/pods/f8238f6d-c793-4f76-b63b-d2d50f9a585d/volumes/kubernetes.io~projected/kube-api-access-cdc27 major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9ccbc38-43f9-4396-bb59-2f746b3d639d/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/f9ccbc38-43f9-4396-bb59-2f746b3d639d/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9ccbc38-43f9-4396-bb59-2f746b3d639d/volumes/kubernetes.io~projected/kube-api-access-vqqbs:{mountpoint:/var/lib/kubelet/pods/f9ccbc38-43f9-4396-bb59-2f746b3d639d/volumes/kubernetes.io~projected/kube-api-access-vqqbs major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9ccbc38-43f9-4396-bb59-2f746b3d639d/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/f9ccbc38-43f9-4396-bb59-2f746b3d639d/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9ccbc38-43f9-4396-bb59-2f746b3d639d/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/f9ccbc38-43f9-4396-bb59-2f746b3d639d/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78/volumes/kubernetes.io~projected/kube-api-access-b7854:{mountpoint:/var/lib/kubelet/pods/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78/volumes/kubernetes.io~projected/kube-api-access-b7854 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78/volumes/kubernetes.io~secret/default-certificate major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78/volumes/kubernetes.io~secret/stats-auth major:0 minor:197 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/70065b202a2a8837a11f1e9fea72fc00c559822cbb70566e1a6e3d1a00ad4abd/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/fa42d44979f132b1d3ea0b2b82be8d2a36a548b261edb970d57cc4bcdd5212c1/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/831e779303cf95a05957826133c169f18129a4eff8464fbd4e4581ca23b42f02/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/687d69a56ab41d0457b059638c9b3699476318af543ec4b60ad613b28dc10b27/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/0079616d34bcc17e036de3da330417ed8d1ac58df8f7471fcea83019ab1bd274/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/c386d5c67ca07659f216b5226e55ebe3793a15537b16d904d2234067b3bac6cf/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/a4725c29223565d1fe9cb1dce8f7bd153b4bdae01cd42a4545b786c6b284d078/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/3e91cef68b569b0251d860f7ca2969e558da96b117504579539fd35988dab0c4/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/89863078d313b0b37c1dd779a82741b1ef73fe665f7a6f8f5e4f2cc1c9d8a512/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/b4680c172a22d626c36f77a91ea4d0d22f1bc80724b0a4e387c710b8b7db9708/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/cea3cb5277a6c0fa4fa89cc7b4a3d4f5d87ceae77aefb131e2241b6c4af1f4b4/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/1856c4fcdfe6933989df5545a8953176abe5c41ac8db9c026a00134e161eb6ee/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/f730c7d80e269d05b80dbb05f7446219d6e3a1fae094a1bf3fca8bd9ff2d6fd0/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/6a78ed83389e5a16895b37bbf1591f4d05095f932be384d74a1d87ccab9caff6/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/1a81a8d5e0e275abbe845b73478f51e27da2da2e5ebac17d70cd2ada90a94011/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/c04e6d2985cef4b54a72cbc5c3b801ee94970ede38d6188c65140fec49bd4657/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/36db1befc23ebed273c53b7c10b8830350eb05a9b35f8619bc3e3f3c343ceeb5/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/7f607f6d8c0d6d1ecb70f2945df9d0e4eb134150ca0b5884ab9ceedf82f758b0/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/b7d913375f3eb6bfb96a4a35babd2f9cbf3fcf1145b25548c24d219dbe9b7d2f/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/ff7fba6c11d51296c5fe8030726f48b9b234cbe96f1ae6a3d832f5f79dbad9d5/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/2dd5cd65dcab015483df8e491f03ca799c66b2edc2062a88546b0d2180142eab/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/f884519f10772fce0c1b1db8568171911962a46a2cbcf69c11d605305158db15/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/a010d6a8fbaee76e6a9133a057c90093f29eddf7a5b8d9e377beab11f85c0040/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/bd67fbe098592ae8a001e9fa3ce915dc7323075ca28b8c1c2e0a6597eb1bc301/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/5ed8b57398046c73643846d044a8da7b98a28c30400ceb81ed919a34da8bbe6b/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/b5059fceb19b314d7441b1b45d6410b097706e0f6992daf2d7e85437ef4bc2f8/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/c8a307cb94fc7e56ce65743f856286a9bacc8ea8b45359d6e88a2cda5c24f7b6/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/e52ab88a03af19ed533603e115f68ad3b35bade142b9c9f4a5ecaee241c013e6/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/cf69d10a3ba1019837988a08d476a33e6d6054b03606d7ce837ce86a3bda223b/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/a6a002d5fd7fe5d5f76036ef4346c39c5aee64c225ad06a76bf140af0783e657/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/5db6d1a49ebd9192d136bb7a69ae388dbe1beae04438485614575ee0c64cd788/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/1d090141fe4b2bc12d04ea117588acbe175ef68d10b7cea87788a5a989b4caeb/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/6190e878fa83d2ca2929d532e556bcf97a816afc9df013d1d8895ef636dca404/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/a9d26ebba77b62092f9ad57295ad82d0c5e3216735002090afcbdeb72eae5772/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/fadb906632acdce9e7c1a36dd2b2403522431b534bcc5757ec059673137f8b96/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/5b4a91479f263f34431b2d154eba3668ca39535a420bbdc2e6981d24e611765c/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/d3e51ffcfd9d0799f21f7749483b43c04d1b6cf037d1d6673c732cb7269d822c/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/f58939b5e18be6b78e2711721dfcf9f20d515e5b828509e69f619974531c8f15/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/b335966ed9805de58c9c85bbf41fd5dae5c5f26a86a4ac080dee8c84bd2de0a7/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/03d4d6b3620ba3733845cc82417750c60377d3ba64ccd4d96c35731e5e5ea876/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/b30651fab13aef47f6aedfb6942a425cf484bdbfb480995ca1e927098ec8d624/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/7de98d71a9badc4c97fa70d4d0f8b8f3fcb35ef2fc1b6a5b546021571bc7e4d1/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/4012640d5ec25f7dbeccc1f5d0493819a8b0b408bd25ae670f7191ca4df4c6cd/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/ad5e62af51b238669c89ca3958ba055833a1f08a880eb4e74f3272fa2d846151/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/04f26cd306e8bee122376ac689a2d7d73a9700cd1b36ee253024e4fb08efbb6d/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/f55341731aed4a82a0f066b3d586dda4f398129708221c7e8fe88adbf5ea0e79/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/0fbc065965ae2b4ac4eb06e4be563dc14d6efbb166d7e15a322208c5a3993503/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/4341499ca548e477d48bbaec86def62598e33d14fc39bc39f3b3974d3589d5e8/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/497bc817dc7f051156b7bf97459ff57cb53c547cda165c5bdfea47818a972607/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/cb03654bc7e081995895a0ca21758d25d9f93fb237658410e8c19df3481b23fc/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/9c65e53d1cc077a180b41157216e561de0fb2c450d980b0de21fe93e6658a0e7/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/d5007f340e2e9a5d7ffaa7e169f24b7aa4a6aee1ccc1ce7b36523aa1dd9d6f69/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/e731a337bffc867ff15660ee69f816aea1ff56e304080749a06a38c0112e29c8/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/a38b5f2ac733b55d27cd0137b97a8cfb793ae1004591161078115100a207a8de/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/a8ff109879bde2e7527c8a3494e8ff4bbb0ca0b3a028f8191d0e65bae3873815/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/8ad5e16d819bf17bf61b0ed9e5dd20b3e35d1846abf0b482d6d2e9538334a0c7/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/431786eec5f9eeadcef51fd951947d694b5c260295c792005756d4ebd979e15f/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/5420a5c59d693ecd6f9dc7dfced9a87111de75d7392e9c3c44326a31f5ccbf93/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/627ddd9f36ea8d0fec42397a7996f3dd35452f91a4132762528478615cc679a8/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/16927a37b5c08abd3c9fc42ac041019c863a39e8f84ab1b16ad2c84b6b8c5ebd/merged major:0 minor:98 fsType:overlay blockSize:0}] Nov 01 05:45:21.445217 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.439873 5040 manager.go:217] Machine: {Timestamp:2026-11-01 05:45:21.439075741 +0000 UTC m=+0.094192899 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100179 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec249910224aa2388d41d80fa895cfb4 SystemUUID:ec249910-224a-a238-8d41-d80fa895cfb4 BootID:2f0bdd67-682e-41f4-a088-ce025a98d64e Filesystems:[{Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/293a6ef2-f0c9-4390-ac82-028792a3b64d/volumes/kubernetes.io~projected/kube-api-access-6pdtj DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c708ffb6309a237f6a87e0ee4823bdacca0cf612f61acbbf7fc5907afbbd4c1e/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/293a6ef2-f0c9-4390-ac82-028792a3b64d/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f9ccbc38-43f9-4396-bb59-2f746b3d639d/volumes/kubernetes.io~projected/kube-api-access-vqqbs DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7729d415-70e5-4be3-a9cc-bbd13e065c71/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/109327c0-f8f8-4dde-9cfe-489bf5a7eaad/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/109327c0-f8f8-4dde-9cfe-489bf5a7eaad/volumes/kubernetes.io~projected/kube-api-access-j64kh DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1b19c3b0-887e-4e27-bac3-028003f1aac5/volumes/kubernetes.io~projected/kube-api-access-fzj6t DeviceMajor:0 DeviceMinor:233 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/baad0c0a-7a5d-454d-8ea9-4a3b27115992/volumes/kubernetes.io~projected/kube-api-access-kqxsx DeviceMajor:0 DeviceMinor:345 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0ac682e4e0215609e5bd32a4d45b55c4e7f9e2fe58ba5bf16998f3978c5016aa/userdata/shm DeviceMajor:0 DeviceMinor:347 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/21519345-86b8-441e-a395-1e8385cd3732/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/580bebeb-9495-4541-8d1d-3bf6fef1a061/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f5705a40-d3e4-40a8-a561-17c435f388b1/volumes/kubernetes.io~projected/kube-api-access-dzxml DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/31894c07-ba1e-4865-9481-58555f919bcc/volumes/kubernetes.io~projected/kube-api-access-t6r8g DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f9ccbc38-43f9-4396-bb59-2f746b3d639d/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2191a9b0-b911-46e9-8ed4-b5119e37341d/volumes/kubernetes.io~projected/kube-api-access-b4v9k DeviceMajor:0 DeviceMinor:327 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7db79928-e03b-4027-9cca-40191f6a300d/volumes/kubernetes.io~projected/kube-api-access-jmjjx DeviceMajor:0 DeviceMinor:340 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/de981773-b03b-4332-b2c1-b8d894423c57/volumes/kubernetes.io~projected/kube-api-access-mxn5f DeviceMajor:0 DeviceMinor:361 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/40592ff5d356deee21ad5769e9533ff695bdf1eb214a9155399cf35ce47382eb/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/21519345-86b8-441e-a395-1e8385cd3732/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/f4051a42-5120-4b67-b904-5db6b4897e07/volumes/kubernetes.io~projected/kube-api-access-vgrrf DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7729d415-70e5-4be3-a9cc-bbd13e065c71/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:227 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6cd67a78-09f1-4988-99b0-1ac9097f2f39/volumes/kubernetes.io~projected/kube-api-access-kz6tw DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f9ccbc38-43f9-4396-bb59-2f746b3d639d/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7729d415-70e5-4be3-a9cc-bbd13e065c71/volumes/kubernetes.io~projected/kube-api-access-72qwb DeviceMajor:0 DeviceMinor:230 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7dc7486d-3188-4bb8-b907-b653caa18660/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:282 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a3c54009ff4e69efff07e45faa1e917b0aec9d2b2f3fd61935987368b12409c5/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6d7d36ac248e6bfb113ab4cda9fbc9207bd58b97e154ecad53c594f39f56d265/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/944070a7add267d0399adb6828fa1d1240081b78d60309c0db13f195a30023b9/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3b51306f82e344f59688bfb3a69b3ac0b052bad3fd4178b3b55cbba30d25e728/userdata/shm DeviceMajor:0 DeviceMinor:75 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1cc9d6f4-202c-45bb-8a4c-4b8e936874de/volumes/kubernetes.io~projected/kube-api-access-n2nbg DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f8238f6d-c793-4f76-b63b-d2d50f9a585d/volumes/kubernetes.io~projected/kube-api-access-cdc27 DeviceMajor:0 DeviceMinor:232 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0719dc90-dbae-425a-8439-812425e1b54c/volumes/kubernetes.io~projected/kube-api-access-mdrd9 DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8/volumes/kubernetes.io~projected/kube-api-access-rrtw2 DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cbc9c127-21a0-4020-a093-d02b4061db5a/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/72f98900-ff17-447f-8429-60c3e47fc7d1/volumes/kubernetes.io~projected/kube-api-access-r5rfj DeviceMajor:0 DeviceMinor:231 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ab197e3ba9a003c80fb73bbbbbf29e0a72c041bd2197a8e41ecf584a1ea2bb59/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d9e9d026b79887b4ae830a3a92925b64430b60f6be59cfe1ea96269f13a49c92/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/529e2d97-b167-4011-a215-02ece8519c00/volumes/kubernetes.io~projected/kube-api-access-k2xnh DeviceMajor:0 DeviceMinor:287 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7729d415-70e5-4be3-a9cc-bbd13e065c71/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:234 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5830b534fa957f41baf2438317c7ba100f3185587842dc614d50d59283bb2b03/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/21519345-86b8-441e-a395-1e8385cd3732/volumes/kubernetes.io~projected/kube-api-access-g6xv7 DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a2520662d22589e5228afa64302b95d581c324972c73bed110953599201ea391/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/71a7ebb53a171eaa9b0ddeaf8dd3dc71aef2a8ee82783e0e1ef4e04e7164a4d0/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/18a61660adc8698ee2d89530b878234ff5b08430b0da524a730460af2bbd2fb0/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78/volumes/kubernetes.io~projected/kube-api-access-b7854 DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/92863f6fea65c046e369857501b96b2651633f50cf45cf0235da0464816efc46/userdata/shm DeviceMajor:0 DeviceMinor:212 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/63c68b10a5fb373166458b336ce3d061b5ac927911f63e8fee1de07f8f8ce9dc/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/197fb18c-6703-4deb-93b3-0186d7ceed67/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dc9422c2-b041-4f6b-8495-961a9571c1e4/volumes/kubernetes.io~projected/kube-api-access-jsp9k DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5e70277c-fae5-4e5a-b3d0-743d7485d2b3/volumes/kubernetes.io~projected/kube-api-access-swfdf DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cb0594fd3b22c52995ef1885c06f2ce591847c1b6c7cc9f8a992e1b203bdddf9/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/39716ebe9bcbd9040dc45c45dc40721cf58f9d698fbb5de683fe2c1b765ab803/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2/volumes/kubernetes.io~projected/kube-api-access-2lmfd DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9fd52c62387ec79eddc2fc1c21b7174cd35bdb356617695d27af2b5a992b9442/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f9ccbc38-43f9-4396-bb59-2f746b3d639d/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/580bebeb-9495-4541-8d1d-3bf6fef1a061/volumes/kubernetes.io~projected/kube-api-access-mk8zm DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/31eaa37e8e6c221992305c756157e9d9398507aa46ab6c8bf70320aaf77c31ce/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/01d6bddcc9f3f9f4ef04ba08d2f3a3f461fa75f8f945c2ca62706c43440ec750/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/47088d4b-2aa6-4f69-9847-f361e4fa5f7a/volumes/kubernetes.io~projected/kube-api-access-5gcvx DeviceMajor:0 DeviceMinor:322 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1b19c3b0-887e-4e27-bac3-028003f1aac5/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:226 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b483fa0870b72cbf1958ccb1c0d87f87e3957d4468bf8eba5fb0fa2a887831b4/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0f3e2e028ef2253cf0b2d40ba7a04a569991030505557c23570fea3e72cd9bb6/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cbc9c127-21a0-4020-a093-d02b4061db5a/volumes/kubernetes.io~projected/kube-api-access-6hksk DeviceMajor:0 DeviceMinor:235 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a9dd32ace74afdc7105a4060ccacc954ea3c04bed453244c8496e0745d835782/userdata/shm DeviceMajor:0 DeviceMinor:335 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/de981773-b03b-4332-b2c1-b8d894423c57/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:360 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2acbc647-2361-4935-b503-5bd9af74f822/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cde21439-a494-47cd-8cc5-161fc96a6701/volumes/kubernetes.io~projected/kube-api-access-8dbh2 DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf8defe53821b5fe41a5e5da1523fee6fa713a3253535d5f90f74b6afe37c04c/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2acbc647-2361-4935-b503-5bd9af74f822/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2acbc647-2361-4935-b503-5bd9af74f822/volumes/kubernetes.io~projected/kube-api-access-4s56k DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a1f5a77d-43d5-48bc-a8b5-5db8f1152853/volumes/kubernetes.io~projected/kube-api-access-f8fz2 DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b6793ef1d065f211ddb1620589da679fd99193c116c959999b841959fce9055/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/67ae987f-79b9-4852-98bd-e0537c412b17/volumes/kubernetes.io~projected/kube-api-access-7gbgq DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/746049f9b24a68fedc050b50f4332cafda0a4d614d8feaa7cc941e321dcf62c9/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:01d6bddcc9f3f9f MacAddress:22:75:50:86:31:5c Speed:10000 Mtu:8901} {Name:0ac682e4e021560 MacAddress:56:ae:e4:ef:91:3b Speed:10000 Mtu:8901} {Name:31eaa37e8e6c221 MacAddress:d2:67:bd:d4:5d:86 Speed:10000 Mtu:8901} {Name:39716ebe9bcbd90 MacAddress:02:03:ce:eb:e1:a8 Speed:10000 Mtu:8901} {Name:4b6793ef1d065f2 MacAddress:46:2a:0b:18:d9:d0 Speed:10000 Mtu:8901} {Name:5830b534fa957f4 MacAddress:22:78:0c:ef:76:c8 Speed:10000 Mtu:8901} {Name:63c68b10a5fb373 MacAddress:6e:9b:92:3e:57:bc Speed:10000 Mtu:8901} {Name:71a7ebb53a171ea MacAddress:0a:a9:eb:81:90:1f Speed:10000 Mtu:8901} {Name:92863f6fea65c04 MacAddress:12:4f:6c:6c:71:8b Speed:10000 Mtu:8901} {Name:a2520662d22589e MacAddress:42:5b:3f:4c:3b:06 Speed:10000 Mtu:8901} {Name:a9dd32ace74afdc MacAddress:42:52:5f:b2:f4:ab Speed:10000 Mtu:8901} {Name:ab197e3ba9a003c MacAddress:6e:85:8f:9c:e9:4e Speed:10000 Mtu:8901} {Name:b483fa0870b72cb MacAddress:96:43:2a:fc:0b:c5 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:4f:1d:06:8d:9d Speed:0 Mtu:9001} {Name:br-int MacAddress:3a:ab:b0:08:92:3f Speed:0 Mtu:8901} {Name:c708ffb6309a237 MacAddress:12:c6:5c:5d:56:bc Speed:10000 Mtu:8901} {Name:cb0594fd3b22c52 MacAddress:a2:4c:79:55:c1:fd Speed:10000 Mtu:8901} {Name:d9e9d026b79887b MacAddress:22:9b:a2:1c:c6:c6 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:4f:1d:06:8d:9d Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:da:25:c7:b0:40:4e Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:2a:14:77:84:70:75 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 05:45:21.445217 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.440868 5040 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 05:45:21.445217 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.440949 5040 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 05:45:21.445217 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.441562 5040 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 05:45:21.445217 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.441588 5040 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-195.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 05:45:21.445217 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.441765 5040 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 05:45:21.445217 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.441776 5040 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 05:45:21.445217 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.441791 5040 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 05:45:21.445217 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.441815 5040 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 05:45:21.445217 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.441836 5040 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 05:45:21.445217 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.442035 5040 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 05:45:21.445217 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.442163 5040 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 05:45:21.445217 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.442204 5040 kubelet.go:493] "Attempting to sync node with API server" Nov 01 05:45:21.445217 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.442214 5040 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 05:45:21.445217 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.442224 5040 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 05:45:21.445217 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.442233 5040 kubelet.go:405] "Adding apiserver pod source" Nov 01 05:45:21.445217 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.442241 5040 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 05:45:21.445217 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.443018 5040 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 05:45:21.445217 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.443029 5040 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 05:45:21.445217 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.443591 5040 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 05:45:21.458704 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.458682 5040 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 05:45:21.458704 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.458700 5040 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 05:45:21.458915 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.458839 5040 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 01 05:45:21.459139 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.459122 5040 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 05:45:21.459231 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.459165 5040 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 05:45:21.459310 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.459298 5040 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 05:45:21.459364 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.459319 5040 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 05:45:21.459364 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.459331 5040 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 05:45:21.459364 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.459340 5040 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 05:45:21.459364 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.459349 5040 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 05:45:21.459548 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.459365 5040 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 05:45:21.459548 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.459375 5040 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 05:45:21.459548 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.459385 5040 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 05:45:21.459548 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.459396 5040 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 05:45:21.459548 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.459406 5040 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 05:45:21.459548 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.459426 5040 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 05:45:21.459548 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.459441 5040 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 05:45:21.459548 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.459490 5040 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 05:45:21.459548 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.459500 5040 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 05:45:21.464130 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.463306 5040 server.go:1263] "Started kubelet" Nov 01 05:45:21.464130 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.463655 5040 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 05:45:21.464130 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.463755 5040 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 05:45:21.464130 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.463748 5040 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 05:45:21.464130 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.464073 5040 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 05:45:21.463887 ip-10-0-143-195 systemd[1]: Started Kubernetes Kubelet. Nov 01 05:45:21.465270 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.465116 5040 server.go:317] "Adding debug handlers to kubelet server" Nov 01 05:45:21.470007 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.469983 5040 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 05:45:21.470758 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:21.470608 5040 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-195.us-west-1.compute.internal\" not found" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.471110 5040 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.471142 5040 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.471232 5040 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.471250 5040 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.472640 5040 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 05:39:40 +0000 UTC" deadline="2028-08-16 13:57:19.78485731 +0000 UTC" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.472685 5040 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15704h11m58.312177001s" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.472750 5040 factory.go:55] Registering systemd factory Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.472767 5040 factory.go:223] Registration of the systemd container factory successfully Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.472982 5040 factory.go:153] Registering CRI-O factory Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.472997 5040 factory.go:223] Registration of the crio container factory successfully Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.473053 5040 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.473073 5040 factory.go:103] Registering Raw factory Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.473099 5040 manager.go:1199] Started watching for new ooms in manager Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.473453 5040 manager.go:319] Starting recovery of all containers Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.474683 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbc9c127-21a0-4020-a093-d02b4061db5a" volumeName="kubernetes.io/configmap/cbc9c127-21a0-4020-a093-d02b4061db5a-service-ca-bundle" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.474718 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbc9c127-21a0-4020-a093-d02b4061db5a" volumeName="kubernetes.io/configmap/cbc9c127-21a0-4020-a093-d02b4061db5a-trusted-ca-bundle" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.474735 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9ccbc38-43f9-4396-bb59-2f746b3d639d" volumeName="kubernetes.io/configmap/f9ccbc38-43f9-4396-bb59-2f746b3d639d-trusted-ca" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.474750 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="109327c0-f8f8-4dde-9cfe-489bf5a7eaad" volumeName="kubernetes.io/projected/109327c0-f8f8-4dde-9cfe-489bf5a7eaad-kube-api-access-j64kh" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.474762 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b19c3b0-887e-4e27-bac3-028003f1aac5" volumeName="kubernetes.io/projected/1b19c3b0-887e-4e27-bac3-028003f1aac5-kube-api-access-fzj6t" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.474774 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbc9c127-21a0-4020-a093-d02b4061db5a" volumeName="kubernetes.io/empty-dir/cbc9c127-21a0-4020-a093-d02b4061db5a-snapshots" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.474800 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9ccbc38-43f9-4396-bb59-2f746b3d639d" volumeName="kubernetes.io/secret/f9ccbc38-43f9-4396-bb59-2f746b3d639d-image-registry-private-configuration" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.474817 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21519345-86b8-441e-a395-1e8385cd3732" volumeName="kubernetes.io/configmap/21519345-86b8-441e-a395-1e8385cd3732-ovnkube-script-lib" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.474833 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2" volumeName="kubernetes.io/configmap/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-samples-operator-config" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.474847 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="529e2d97-b167-4011-a215-02ece8519c00" volumeName="kubernetes.io/projected/529e2d97-b167-4011-a215-02ece8519c00-kube-api-access-k2xnh" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.474878 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="580bebeb-9495-4541-8d1d-3bf6fef1a061" volumeName="kubernetes.io/configmap/580bebeb-9495-4541-8d1d-3bf6fef1a061-trusted-ca" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.474890 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72f98900-ff17-447f-8429-60c3e47fc7d1" volumeName="kubernetes.io/projected/72f98900-ff17-447f-8429-60c3e47fc7d1-kube-api-access-r5rfj" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.474902 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbc9c127-21a0-4020-a093-d02b4061db5a" volumeName="kubernetes.io/empty-dir/cbc9c127-21a0-4020-a093-d02b4061db5a-tmp" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.474916 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8" volumeName="kubernetes.io/empty-dir/ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8-tmp-dir" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.474932 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5705a40-d3e4-40a8-a561-17c435f388b1" volumeName="kubernetes.io/configmap/f5705a40-d3e4-40a8-a561-17c435f388b1-cni-sysctl-allowlist" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.474948 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0719dc90-dbae-425a-8439-812425e1b54c" volumeName="kubernetes.io/configmap/0719dc90-dbae-425a-8439-812425e1b54c-iptables-alerter-script" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.474960 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="293a6ef2-f0c9-4390-ac82-028792a3b64d" volumeName="kubernetes.io/secret/293a6ef2-f0c9-4390-ac82-028792a3b64d-serving-cert" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.474971 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47088d4b-2aa6-4f69-9847-f361e4fa5f7a" volumeName="kubernetes.io/configmap/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-trusted-ca-bundle" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.474985 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47088d4b-2aa6-4f69-9847-f361e4fa5f7a" volumeName="kubernetes.io/projected/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-kube-api-access-5gcvx" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.474999 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21519345-86b8-441e-a395-1e8385cd3732" volumeName="kubernetes.io/configmap/21519345-86b8-441e-a395-1e8385cd3732-env-overrides" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475012 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="580bebeb-9495-4541-8d1d-3bf6fef1a061" volumeName="kubernetes.io/configmap/580bebeb-9495-4541-8d1d-3bf6fef1a061-config" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475028 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7729d415-70e5-4be3-a9cc-bbd13e065c71" volumeName="kubernetes.io/empty-dir/7729d415-70e5-4be3-a9cc-bbd13e065c71-ca-trust-extracted" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475040 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbc9c127-21a0-4020-a093-d02b4061db5a" volumeName="kubernetes.io/secret/cbc9c127-21a0-4020-a093-d02b4061db5a-serving-cert" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475054 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9ccbc38-43f9-4396-bb59-2f746b3d639d" volumeName="kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-bound-sa-token" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475066 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78" volumeName="kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-default-certificate" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475078 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9ccbc38-43f9-4396-bb59-2f746b3d639d" volumeName="kubernetes.io/configmap/f9ccbc38-43f9-4396-bb59-2f746b3d639d-registry-certificates" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475120 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78" volumeName="kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-stats-auth" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475134 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6cd67a78-09f1-4988-99b0-1ac9097f2f39" volumeName="kubernetes.io/projected/6cd67a78-09f1-4988-99b0-1ac9097f2f39-kube-api-access-kz6tw" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475151 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbc9c127-21a0-4020-a093-d02b4061db5a" volumeName="kubernetes.io/projected/cbc9c127-21a0-4020-a093-d02b4061db5a-kube-api-access-6hksk" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475182 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5705a40-d3e4-40a8-a561-17c435f388b1" volumeName="kubernetes.io/configmap/f5705a40-d3e4-40a8-a561-17c435f388b1-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475197 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7729d415-70e5-4be3-a9cc-bbd13e065c71" volumeName="kubernetes.io/secret/7729d415-70e5-4be3-a9cc-bbd13e065c71-installation-pull-secrets" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475209 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="109327c0-f8f8-4dde-9cfe-489bf5a7eaad" volumeName="kubernetes.io/secret/109327c0-f8f8-4dde-9cfe-489bf5a7eaad-serving-cert" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475220 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="375debb4-cfe1-44ab-907b-0a34a031c2ea" volumeName="kubernetes.io/configmap/375debb4-cfe1-44ab-907b-0a34a031c2ea-nginx-conf" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475233 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7729d415-70e5-4be3-a9cc-bbd13e065c71" volumeName="kubernetes.io/configmap/7729d415-70e5-4be3-a9cc-bbd13e065c71-registry-certificates" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475247 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9ccbc38-43f9-4396-bb59-2f746b3d639d" volumeName="kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-kube-api-access-vqqbs" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475261 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9ccbc38-43f9-4396-bb59-2f746b3d639d" volumeName="kubernetes.io/secret/f9ccbc38-43f9-4396-bb59-2f746b3d639d-installation-pull-secrets" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475281 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7dc7486d-3188-4bb8-b907-b653caa18660" volumeName="kubernetes.io/secret/7dc7486d-3188-4bb8-b907-b653caa18660-original-pull-secret" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475292 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1cc9d6f4-202c-45bb-8a4c-4b8e936874de" volumeName="kubernetes.io/projected/1cc9d6f4-202c-45bb-8a4c-4b8e936874de-kube-api-access-n2nbg" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475304 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21519345-86b8-441e-a395-1e8385cd3732" volumeName="kubernetes.io/projected/21519345-86b8-441e-a395-1e8385cd3732-kube-api-access-g6xv7" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475318 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2191a9b0-b911-46e9-8ed4-b5119e37341d" volumeName="kubernetes.io/empty-dir/2191a9b0-b911-46e9-8ed4-b5119e37341d-data-volume" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475345 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2acbc647-2361-4935-b503-5bd9af74f822" volumeName="kubernetes.io/empty-dir/2acbc647-2361-4935-b503-5bd9af74f822-tmp" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475359 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5705a40-d3e4-40a8-a561-17c435f388b1" volumeName="kubernetes.io/projected/f5705a40-d3e4-40a8-a561-17c435f388b1-kube-api-access-dzxml" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475374 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b19c3b0-887e-4e27-bac3-028003f1aac5" volumeName="kubernetes.io/secret/1b19c3b0-887e-4e27-bac3-028003f1aac5-cert" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475387 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="293a6ef2-f0c9-4390-ac82-028792a3b64d" volumeName="kubernetes.io/projected/293a6ef2-f0c9-4390-ac82-028792a3b64d-kube-api-access-6pdtj" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475400 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47088d4b-2aa6-4f69-9847-f361e4fa5f7a" volumeName="kubernetes.io/configmap/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-service-ca-bundle" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475438 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67ae987f-79b9-4852-98bd-e0537c412b17" volumeName="kubernetes.io/projected/67ae987f-79b9-4852-98bd-e0537c412b17-kube-api-access-7gbgq" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475453 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72f98900-ff17-447f-8429-60c3e47fc7d1" volumeName="kubernetes.io/configmap/72f98900-ff17-447f-8429-60c3e47fc7d1-config-volume" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475467 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7729d415-70e5-4be3-a9cc-bbd13e065c71" volumeName="kubernetes.io/configmap/7729d415-70e5-4be3-a9cc-bbd13e065c71-trusted-ca" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475496 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc9422c2-b041-4f6b-8495-961a9571c1e4" volumeName="kubernetes.io/configmap/dc9422c2-b041-4f6b-8495-961a9571c1e4-telemetry-config" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475509 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8" volumeName="kubernetes.io/projected/ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8-kube-api-access-rrtw2" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475523 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7729d415-70e5-4be3-a9cc-bbd13e065c71" volumeName="kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-bound-sa-token" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475538 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7db79928-e03b-4027-9cca-40191f6a300d" volumeName="kubernetes.io/projected/7db79928-e03b-4027-9cca-40191f6a300d-kube-api-access-jmjjx" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475551 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1f5a77d-43d5-48bc-a8b5-5db8f1152853" volumeName="kubernetes.io/projected/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-kube-api-access-f8fz2" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475563 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21519345-86b8-441e-a395-1e8385cd3732" volumeName="kubernetes.io/secret/21519345-86b8-441e-a395-1e8385cd3732-ovn-node-metrics-cert" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475574 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cde21439-a494-47cd-8cc5-161fc96a6701" volumeName="kubernetes.io/configmap/cde21439-a494-47cd-8cc5-161fc96a6701-serviceca" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475587 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de981773-b03b-4332-b2c1-b8d894423c57" volumeName="kubernetes.io/projected/de981773-b03b-4332-b2c1-b8d894423c57-kube-api-access-mxn5f" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475610 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de981773-b03b-4332-b2c1-b8d894423c57" volumeName="kubernetes.io/secret/de981773-b03b-4332-b2c1-b8d894423c57-signing-key" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475621 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4051a42-5120-4b67-b904-5db6b4897e07" volumeName="kubernetes.io/projected/f4051a42-5120-4b67-b904-5db6b4897e07-kube-api-access-vgrrf" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475634 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2191a9b0-b911-46e9-8ed4-b5119e37341d" volumeName="kubernetes.io/projected/2191a9b0-b911-46e9-8ed4-b5119e37341d-kube-api-access-b4v9k" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475645 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc9422c2-b041-4f6b-8495-961a9571c1e4" volumeName="kubernetes.io/projected/dc9422c2-b041-4f6b-8495-961a9571c1e4-kube-api-access-jsp9k" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475656 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9ccbc38-43f9-4396-bb59-2f746b3d639d" volumeName="kubernetes.io/empty-dir/f9ccbc38-43f9-4396-bb59-2f746b3d639d-ca-trust-extracted" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475666 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78" volumeName="kubernetes.io/projected/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-kube-api-access-b7854" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475678 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="197fb18c-6703-4deb-93b3-0186d7ceed67" volumeName="kubernetes.io/secret/197fb18c-6703-4deb-93b3-0186d7ceed67-agent-certs" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475692 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="580bebeb-9495-4541-8d1d-3bf6fef1a061" volumeName="kubernetes.io/projected/580bebeb-9495-4541-8d1d-3bf6fef1a061-kube-api-access-mk8zm" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475703 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cde21439-a494-47cd-8cc5-161fc96a6701" volumeName="kubernetes.io/projected/cde21439-a494-47cd-8cc5-161fc96a6701-kube-api-access-8dbh2" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475714 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5705a40-d3e4-40a8-a561-17c435f388b1" volumeName="kubernetes.io/configmap/f5705a40-d3e4-40a8-a561-17c435f388b1-cni-binary-copy" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475727 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8238f6d-c793-4f76-b63b-d2d50f9a585d" volumeName="kubernetes.io/projected/f8238f6d-c793-4f76-b63b-d2d50f9a585d-kube-api-access-cdc27" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475738 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2acbc647-2361-4935-b503-5bd9af74f822" volumeName="kubernetes.io/empty-dir/2acbc647-2361-4935-b503-5bd9af74f822-etc-tuned" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475749 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2" volumeName="kubernetes.io/projected/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-kube-api-access-2lmfd" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475763 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e70277c-fae5-4e5a-b3d0-743d7485d2b3" volumeName="kubernetes.io/projected/5e70277c-fae5-4e5a-b3d0-743d7485d2b3-kube-api-access-swfdf" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475778 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67ae987f-79b9-4852-98bd-e0537c412b17" volumeName="kubernetes.io/configmap/67ae987f-79b9-4852-98bd-e0537c412b17-cni-binary-copy" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475790 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="109327c0-f8f8-4dde-9cfe-489bf5a7eaad" volumeName="kubernetes.io/configmap/109327c0-f8f8-4dde-9cfe-489bf5a7eaad-config" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475802 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2191a9b0-b911-46e9-8ed4-b5119e37341d" volumeName="kubernetes.io/configmap/2191a9b0-b911-46e9-8ed4-b5119e37341d-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475815 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7729d415-70e5-4be3-a9cc-bbd13e065c71" volumeName="kubernetes.io/secret/7729d415-70e5-4be3-a9cc-bbd13e065c71-image-registry-private-configuration" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475828 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de981773-b03b-4332-b2c1-b8d894423c57" volumeName="kubernetes.io/configmap/de981773-b03b-4332-b2c1-b8d894423c57-signing-cabundle" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475842 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="baad0c0a-7a5d-454d-8ea9-4a3b27115992" volumeName="kubernetes.io/projected/baad0c0a-7a5d-454d-8ea9-4a3b27115992-kube-api-access-kqxsx" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475856 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0719dc90-dbae-425a-8439-812425e1b54c" volumeName="kubernetes.io/projected/0719dc90-dbae-425a-8439-812425e1b54c-kube-api-access-mdrd9" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475866 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="293a6ef2-f0c9-4390-ac82-028792a3b64d" volumeName="kubernetes.io/configmap/293a6ef2-f0c9-4390-ac82-028792a3b64d-config" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475875 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2acbc647-2361-4935-b503-5bd9af74f822" volumeName="kubernetes.io/projected/2acbc647-2361-4935-b503-5bd9af74f822-kube-api-access-4s56k" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475884 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47088d4b-2aa6-4f69-9847-f361e4fa5f7a" volumeName="kubernetes.io/empty-dir/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-archives-path" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475896 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="580bebeb-9495-4541-8d1d-3bf6fef1a061" volumeName="kubernetes.io/secret/580bebeb-9495-4541-8d1d-3bf6fef1a061-serving-cert" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475910 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7729d415-70e5-4be3-a9cc-bbd13e065c71" volumeName="kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-kube-api-access-72qwb" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475923 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="197fb18c-6703-4deb-93b3-0186d7ceed67" volumeName="kubernetes.io/configmap/197fb18c-6703-4deb-93b3-0186d7ceed67-konnectivity-ca" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475934 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21519345-86b8-441e-a395-1e8385cd3732" volumeName="kubernetes.io/configmap/21519345-86b8-441e-a395-1e8385cd3732-ovnkube-config" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475947 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31894c07-ba1e-4865-9481-58555f919bcc" volumeName="kubernetes.io/projected/31894c07-ba1e-4865-9481-58555f919bcc-kube-api-access-t6r8g" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475968 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67ae987f-79b9-4852-98bd-e0537c412b17" volumeName="kubernetes.io/configmap/67ae987f-79b9-4852-98bd-e0537c412b17-multus-daemon-config" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.475991 5040 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72f98900-ff17-447f-8429-60c3e47fc7d1" volumeName="kubernetes.io/empty-dir/72f98900-ff17-447f-8429-60c3e47fc7d1-tmp-dir" seLinuxMountContext="" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.476004 5040 reconstruct.go:97] "Volume reconstruction finished" Nov 01 05:45:21.476649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.476011 5040 reconciler.go:29] "Reconciler: start to sync state" Nov 01 05:45:21.480964 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.479155 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/1.log" Nov 01 05:45:21.491423 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.491160 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/1.log" Nov 01 05:45:21.505319 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.505145 5040 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 05:45:21.506660 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.506636 5040 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 05:45:21.506660 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.506663 5040 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 05:45:21.506846 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.506685 5040 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 05:45:21.506846 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:21.506731 5040 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 01 05:45:21.523694 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.523667 5040 generic.go:356] "Generic (PLEG): container finished" podID="f5705a40-d3e4-40a8-a561-17c435f388b1" containerID="1fa60482427c4deb73102cf745c83c2c89010d2bd8bccad0c328aa12a1feee6c" exitCode=0 Nov 01 05:45:21.523694 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.523692 5040 generic.go:356] "Generic (PLEG): container finished" podID="f5705a40-d3e4-40a8-a561-17c435f388b1" containerID="c13cd4a1ae3808cf52b7c8b377944e16df698151c3bfa767b85c0746892d7cca" exitCode=0 Nov 01 05:45:21.523694 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.523702 5040 generic.go:356] "Generic (PLEG): container finished" podID="f5705a40-d3e4-40a8-a561-17c435f388b1" containerID="1801d85bfe506b0c3f992052aa347d3f4c3cb27f80a607e858c0c125402a8ae6" exitCode=0 Nov 01 05:45:21.524011 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.523712 5040 generic.go:356] "Generic (PLEG): container finished" podID="f5705a40-d3e4-40a8-a561-17c435f388b1" containerID="cb0e98cc1cdcfd74db81edd1ce2d1a4b01e6177883ac74f7a85b730ef8e3d160" exitCode=0 Nov 01 05:45:21.524011 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.523719 5040 generic.go:356] "Generic (PLEG): container finished" podID="f5705a40-d3e4-40a8-a561-17c435f388b1" containerID="b23ee54876e3a909f039a31bf3bee7bec29cf7bfcbd5e962789257ddfc904063" exitCode=0 Nov 01 05:45:21.524011 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.523727 5040 generic.go:356] "Generic (PLEG): container finished" podID="f5705a40-d3e4-40a8-a561-17c435f388b1" containerID="f77c4d4e94b3626e04c5958abfcf2031137d599ca921225f62e442f996addec4" exitCode=0 Nov 01 05:45:21.546493 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.546454 5040 generic.go:356] "Generic (PLEG): container finished" podID="ac965f1cfbc1129016d728e686326ed6" containerID="cdd5aa48e73883f04b05c4608010e07f9acc13cc79baa6b38526ce5f538f07b4" exitCode=0 Nov 01 05:45:21.570714 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:21.570689 5040 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-195.us-west-1.compute.internal\" not found" Nov 01 05:45:21.578899 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.578875 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/1.log" Nov 01 05:45:21.579039 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.578905 5040 generic.go:356] "Generic (PLEG): container finished" podID="580bebeb-9495-4541-8d1d-3bf6fef1a061" containerID="aeacbab6a94ca1ae2b5b61836714026e446cbb0d816488b4c6374f81be6410c8" exitCode=255 Nov 01 05:45:21.587627 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.587608 5040 manager.go:324] Recovery completed Nov 01 05:45:21.605930 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.605909 5040 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 05:45:21.606869 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:21.606851 5040 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 01 05:45:21.607399 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.607382 5040 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 05:45:21.607812 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.607784 5040 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-195.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 05:45:21.607918 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.607823 5040 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-195.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 05:45:21.607918 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.607840 5040 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-195.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 05:45:21.610699 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.610682 5040 cpu_manager.go:226] "Starting" policy="none" Nov 01 05:45:21.610699 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.610694 5040 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 05:45:21.610845 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.610694 5040 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 05:45:21.610845 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.610772 5040 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 05:45:21.610952 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.610910 5040 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 01 05:45:21.610952 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.610932 5040 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 01 05:45:21.610952 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.610950 5040 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 01 05:45:21.611049 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.610955 5040 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 01 05:45:21.611049 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.610960 5040 policy_none.go:50] "Start" Nov 01 05:45:21.611049 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.610971 5040 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 05:45:21.611049 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.610984 5040 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 05:45:21.611164 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.611119 5040 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 01 05:45:21.611164 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.611126 5040 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 01 05:45:21.611164 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.611132 5040 policy_none.go:44] "Start" Nov 01 05:45:21.611700 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.611683 5040 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-195.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 05:45:21.611763 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.611715 5040 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-195.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 05:45:21.611763 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.611729 5040 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-195.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 05:45:21.618550 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.618520 5040 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 05:45:21.621138 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.621126 5040 manager.go:341] "Starting Device Plugin manager" Nov 01 05:45:21.621197 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:21.621153 5040 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 05:45:21.621197 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.621163 5040 server.go:87] "Starting device plugin registration server" Nov 01 05:45:21.621435 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.621423 5040 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 05:45:21.621936 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.621436 5040 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 05:45:21.622035 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.621982 5040 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 05:45:21.622143 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.622097 5040 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 05:45:21.622246 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.622151 5040 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 05:45:21.622246 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.622159 5040 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 05:45:21.622336 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.622199 5040 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T05:45:21.622137294Z","UUID":"4f8c3f0e-c872-47f2-8d11-75f33e9dd8de","Handler":null,"Name":"","Endpoint":""} Nov 01 05:45:21.623306 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:21.623286 5040 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 05:45:21.623396 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:21.623320 5040 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-195.us-west-1.compute.internal\" not found" Nov 01 05:45:21.623505 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.623470 5040 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 05:45:21.623608 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.623517 5040 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 05:45:21.657343 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.657318 5040 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 05:45:21.657447 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.657388 5040 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 01 05:45:21.658451 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.658436 5040 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 05:45:21.710725 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.710698 5040 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 05:45:21.722606 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.722577 5040 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 05:45:21.723732 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.723709 5040 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-195.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 05:45:21.723836 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.723744 5040 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-195.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 05:45:21.723836 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.723758 5040 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-195.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 05:45:21.723836 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.723803 5040 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:45:21.807395 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.807341 5040 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-195.us-west-1.compute.internal"] Nov 01 05:45:21.808096 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.808052 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal" event={"ID":"ac965f1cfbc1129016d728e686326ed6","Type":"ContainerStarted","Data":"edfd3bedf5e2f01e7b42ca78b9b03f8b95a3df681b1189d6d058e8e2aca61e21"} Nov 01 05:45:21.808156 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.808090 5040 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:45:21.808156 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.808115 5040 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:45:21.808218 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.808115 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal" event={"ID":"ac965f1cfbc1129016d728e686326ed6","Type":"ContainerDied","Data":"cdd5aa48e73883f04b05c4608010e07f9acc13cc79baa6b38526ce5f538f07b4"} Nov 01 05:45:21.808251 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.808220 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal" event={"ID":"ac965f1cfbc1129016d728e686326ed6","Type":"ContainerStarted","Data":"18a61660adc8698ee2d89530b878234ff5b08430b0da524a730460af2bbd2fb0"} Nov 01 05:45:21.808336 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.808262 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-195.us-west-1.compute.internal" event={"ID":"c656c4baa8463ac8d24886e04ca99821","Type":"ContainerStarted","Data":"f4d5b687ca74a3185a37acbcf44ddff111db767b33ad49551a003d7af8c260aa"} Nov 01 05:45:21.808336 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.808277 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-195.us-west-1.compute.internal" event={"ID":"c656c4baa8463ac8d24886e04ca99821","Type":"ContainerStarted","Data":"9fd52c62387ec79eddc2fc1c21b7174cd35bdb356617695d27af2b5a992b9442"} Nov 01 05:45:21.871819 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.871782 5040 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 05:45:21.872038 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:21.871847 5040 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-195.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:45:21.872038 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.871905 5040 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:45:21.872038 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.872004 5040 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:45:21.872241 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.872179 5040 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 05:45:21.872364 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:21.872248 5040 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:45:21.959041 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.958921 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ac965f1cfbc1129016d728e686326ed6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal\" (UID: \"ac965f1cfbc1129016d728e686326ed6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:45:21.959041 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.959024 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac965f1cfbc1129016d728e686326ed6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal\" (UID: \"ac965f1cfbc1129016d728e686326ed6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:45:21.959291 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:21.959083 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c656c4baa8463ac8d24886e04ca99821-config\") pod \"kube-apiserver-proxy-ip-10-0-143-195.us-west-1.compute.internal\" (UID: \"c656c4baa8463ac8d24886e04ca99821\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:45:22.060107 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.060008 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ac965f1cfbc1129016d728e686326ed6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal\" (UID: \"ac965f1cfbc1129016d728e686326ed6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:45:22.060107 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.060056 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac965f1cfbc1129016d728e686326ed6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal\" (UID: \"ac965f1cfbc1129016d728e686326ed6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:45:22.060107 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.060077 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c656c4baa8463ac8d24886e04ca99821-config\") pod \"kube-apiserver-proxy-ip-10-0-143-195.us-west-1.compute.internal\" (UID: \"c656c4baa8463ac8d24886e04ca99821\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:45:22.060382 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.060122 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c656c4baa8463ac8d24886e04ca99821-config\") pod \"kube-apiserver-proxy-ip-10-0-143-195.us-west-1.compute.internal\" (UID: \"c656c4baa8463ac8d24886e04ca99821\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:45:22.060382 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.060129 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ac965f1cfbc1129016d728e686326ed6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal\" (UID: \"ac965f1cfbc1129016d728e686326ed6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:45:22.060382 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.060131 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac965f1cfbc1129016d728e686326ed6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal\" (UID: \"ac965f1cfbc1129016d728e686326ed6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:45:22.443369 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.443287 5040 apiserver.go:52] "Watching apiserver" Nov 01 05:45:22.588035 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.588002 5040 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:45:22.588210 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.588126 5040 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:45:22.605064 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.605030 5040 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 05:45:22.607359 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.607260 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-n6nns","openshift-service-ca-operator/service-ca-operator-544d749c8c-vr978","openshift-network-operator/iptables-alerter-qbg2s","openshift-multus/multus-additional-cni-plugins-b7rrq","openshift-insights/periodic-gathering-nh7q2-cs6dz","openshift-ingress-canary/ingress-canary-dwklx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x","openshift-dns/node-resolver-2qtwx","openshift-image-registry/node-ca-tw96g","openshift-multus/multus-qh47z","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flt78","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-54lc7","openshift-monitoring/cluster-monitoring-operator-884576c57-9j8j4","openshift-dns/dns-default-hxw5f","openshift-cluster-node-tuning-operator/tuned-cb599","openshift-kube-storage-version-migrator/migrator-66df5d496b-wtlk8","openshift-network-console/networking-console-plugin-86c478bb88-7grdp","openshift-image-registry/image-registry-847d56945f-lt2tl","kube-system/konnectivity-agent-n7mks","openshift-ovn-kubernetes/ovnkube-node-7bgdz","kube-system/kube-apiserver-proxy-ip-10-0-143-195.us-west-1.compute.internal","openshift-ingress/router-default-6c5b5d5bd5-776bc","openshift-service-ca/service-ca-5fffc97bdf-zwlff","kube-system/kas-connection-checker-685895c7c9-b2659","openshift-multus/network-metrics-daemon-tt8gq","kube-system/kas-connection-checker-685895c7c9-b2kmh","kube-system/global-pull-secret-syncer-r5l82","openshift-image-registry/image-registry-5f55979cbb-ngg8z","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95sds","openshift-network-diagnostics/network-check-target-ljcjx","openshift-console-operator/console-operator-7cd56b8dbc-njjn4","openshift-network-diagnostics/network-check-source-6f78d59f99-pgxkq","openshift-insights/insights-operator-7c4986d449-r8n4b","openshift-insights/insights-runtime-extractor-54tr8","kube-system/kas-connection-checker-685895c7c9-ls8v2"] Nov 01 05:45:22.610018 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.609995 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:22.610371 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.610352 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flt78" Nov 01 05:45:22.610722 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.610706 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:22.611340 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.611324 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9j8j4" Nov 01 05:45:22.611959 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.611941 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-7grdp" Nov 01 05:45:22.612552 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.612535 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:45:22.617920 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.617904 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:22.619091 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.619077 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hxw5f" Nov 01 05:45:22.619757 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.619742 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:22.654001 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.653984 5040 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 05:45:22.654114 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:22.654040 5040 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-195.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:45:22.654114 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.653984 5040 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 05:45:22.654222 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:22.654123 5040 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-195.us-west-1.compute.internal" Nov 01 05:45:22.666051 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.666010 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 01 05:45:22.666051 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.666019 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 01 05:45:22.666051 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.666020 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 05:45:22.666051 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.666039 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 01 05:45:22.666255 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.666068 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 01 05:45:22.666255 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.666159 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 01 05:45:22.666255 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.666175 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 01 05:45:22.666453 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.666354 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 01 05:45:22.666897 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.666874 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 01 05:45:22.667276 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.667250 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 01 05:45:22.667378 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.667296 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 01 05:45:22.667494 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.667460 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 01 05:45:22.667494 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.667475 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 01 05:45:22.667599 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.667498 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 01 05:45:22.667599 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.667500 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-g592c\"" Nov 01 05:45:22.667599 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.667511 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p4xg8\"" Nov 01 05:45:22.667599 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.667526 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 01 05:45:22.673487 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.673465 5040 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 05:45:22.704687 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.704614 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 01 05:45:22.704687 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.704628 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 01 05:45:22.704687 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.704666 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7s2ht\"" Nov 01 05:45:22.704687 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.704668 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-mzdd9\"" Nov 01 05:45:22.704907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.704729 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-qxt94\"" Nov 01 05:45:22.706558 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.706546 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 05:45:22.706647 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.706633 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 01 05:45:22.711973 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.711953 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 01 05:45:22.712103 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.711992 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 01 05:45:22.712103 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.712000 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 01 05:45:22.712103 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.712010 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 01 05:45:22.712103 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.712005 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 01 05:45:22.712103 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.712039 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 01 05:45:22.714239 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.714226 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 01 05:45:22.714319 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.714243 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 05:45:22.714319 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.714274 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 05:45:22.714319 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.714290 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 01 05:45:22.714473 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.714439 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 05:45:22.714473 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.714458 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 01 05:45:22.715356 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.715340 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 01 05:45:22.715468 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.715450 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 01 05:45:22.715550 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.715533 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 01 05:45:22.715550 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.715541 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 01 05:45:22.715550 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.715539 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-6j8z2\"" Nov 01 05:45:22.715721 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.715555 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 01 05:45:22.717825 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.717810 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 05:45:22.717825 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.717812 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-9g7mn\"" Nov 01 05:45:22.717952 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.717886 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 05:45:22.717952 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.717926 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 05:45:22.717952 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.717929 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 01 05:45:22.717952 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.717926 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 01 05:45:22.718155 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.718075 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-wvmhq\"" Nov 01 05:45:22.718155 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.718091 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 01 05:45:22.718285 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.718269 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 05:45:22.718345 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.718294 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 05:45:22.718345 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.718305 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 05:45:22.718345 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.718322 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 05:45:22.718345 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.718339 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 01 05:45:22.718345 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.718345 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 05:45:22.718589 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.718355 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 05:45:22.718589 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.718374 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 01 05:45:22.718589 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.718375 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 05:45:22.718589 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.718400 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-v8m4f\"" Nov 01 05:45:22.718589 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.718403 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 05:45:22.718589 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.718418 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 05:45:22.718589 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.718427 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 05:45:22.718589 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.718449 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 05:45:22.718589 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.718453 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 05:45:22.718589 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.718455 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 05:45:22.718589 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.718497 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 05:45:22.718589 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.718509 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 01 05:45:22.718589 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.718537 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 05:45:22.719021 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.718644 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 05:45:22.753581 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.753549 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 05:45:22.755358 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.755341 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jzphk\"" Nov 01 05:45:22.755436 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.755376 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-87mzr\"" Nov 01 05:45:22.757313 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.757295 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 05:45:22.757313 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.757300 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 05:45:22.757417 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.757315 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vbc6d\"" Nov 01 05:45:22.757417 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.757317 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xlcv8\"" Nov 01 05:45:22.762835 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.762819 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-695z8\"" Nov 01 05:45:22.762918 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.762831 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-slash\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.762918 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.762857 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/de981773-b03b-4332-b2c1-b8d894423c57-signing-key\") pod \"service-ca-5fffc97bdf-zwlff\" (UID: \"de981773-b03b-4332-b2c1-b8d894423c57\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zwlff" Nov 01 05:45:22.762918 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.762879 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cde21439-a494-47cd-8cc5-161fc96a6701-serviceca\") pod \"node-ca-tw96g\" (UID: \"cde21439-a494-47cd-8cc5-161fc96a6701\") " pod="openshift-image-registry/node-ca-tw96g" Nov 01 05:45:22.763068 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.762969 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/67ae987f-79b9-4852-98bd-e0537c412b17-multus-daemon-config\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.763068 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.763007 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-host\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.763143 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.763099 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7854\" (UniqueName: \"kubernetes.io/projected/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-kube-api-access-b7854\") pod \"router-default-6c5b5d5bd5-776bc\" (UID: \"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78\") " pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:22.763143 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.763134 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0719dc90-dbae-425a-8439-812425e1b54c-iptables-alerter-script\") pod \"iptables-alerter-qbg2s\" (UID: \"0719dc90-dbae-425a-8439-812425e1b54c\") " pod="openshift-network-operator/iptables-alerter-qbg2s" Nov 01 05:45:22.763223 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.763158 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-system-cni-dir\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.763223 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.763178 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/21519345-86b8-441e-a395-1e8385cd3732-ovnkube-script-lib\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.763309 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.763244 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-archives-path\") pod \"periodic-gathering-nh7q2-cs6dz\" (UID: \"47088d4b-2aa6-4f69-9847-f361e4fa5f7a\") " pod="openshift-insights/periodic-gathering-nh7q2-cs6dz" Nov 01 05:45:22.763309 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.763268 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-service-ca-bundle\") pod \"periodic-gathering-nh7q2-cs6dz\" (UID: \"47088d4b-2aa6-4f69-9847-f361e4fa5f7a\") " pod="openshift-insights/periodic-gathering-nh7q2-cs6dz" Nov 01 05:45:22.763416 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.763317 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cde21439-a494-47cd-8cc5-161fc96a6701-serviceca\") pod \"node-ca-tw96g\" (UID: \"cde21439-a494-47cd-8cc5-161fc96a6701\") " pod="openshift-image-registry/node-ca-tw96g" Nov 01 05:45:22.763416 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.763350 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-archives-path\") pod \"periodic-gathering-nh7q2-cs6dz\" (UID: \"47088d4b-2aa6-4f69-9847-f361e4fa5f7a\") " pod="openshift-insights/periodic-gathering-nh7q2-cs6dz" Nov 01 05:45:22.763416 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.763397 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-service-ca-bundle\") pod \"periodic-gathering-nh7q2-cs6dz\" (UID: \"47088d4b-2aa6-4f69-9847-f361e4fa5f7a\") " pod="openshift-insights/periodic-gathering-nh7q2-cs6dz" Nov 01 05:45:22.763537 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.763416 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cde21439-a494-47cd-8cc5-161fc96a6701-host\") pod \"node-ca-tw96g\" (UID: \"cde21439-a494-47cd-8cc5-161fc96a6701\") " pod="openshift-image-registry/node-ca-tw96g" Nov 01 05:45:22.763537 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.763452 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-var-lib-kubelet\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.763537 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.763493 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j64kh\" (UniqueName: \"kubernetes.io/projected/109327c0-f8f8-4dde-9cfe-489bf5a7eaad-kube-api-access-j64kh\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-95sds\" (UID: \"109327c0-f8f8-4dde-9cfe-489bf5a7eaad\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95sds" Nov 01 05:45:22.763620 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.763600 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-run\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.763654 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.763620 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-host-run-multus-certs\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.763700 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.763646 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-systemd-units\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.763700 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.763670 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2191a9b0-b911-46e9-8ed4-b5119e37341d-data-volume\") pod \"insights-runtime-extractor-54tr8\" (UID: \"2191a9b0-b911-46e9-8ed4-b5119e37341d\") " pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:22.763777 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.763710 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-stats-auth\") pod \"router-default-6c5b5d5bd5-776bc\" (UID: \"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78\") " pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:22.763829 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.763788 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-cnibin\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.763829 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.763802 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2191a9b0-b911-46e9-8ed4-b5119e37341d-data-volume\") pod \"insights-runtime-extractor-54tr8\" (UID: \"2191a9b0-b911-46e9-8ed4-b5119e37341d\") " pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:22.763968 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.763843 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-lib-modules\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.763968 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.763876 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-metrics-certs\") pod \"router-default-6c5b5d5bd5-776bc\" (UID: \"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78\") " pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:22.763968 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.763908 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-stats-auth\") pod \"router-default-6c5b5d5bd5-776bc\" (UID: \"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78\") " pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:22.763968 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.763912 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7729d415-70e5-4be3-a9cc-bbd13e065c71-ca-trust-extracted\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:22.763968 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.763955 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/580bebeb-9495-4541-8d1d-3bf6fef1a061-trusted-ca\") pod \"console-operator-7cd56b8dbc-njjn4\" (UID: \"580bebeb-9495-4541-8d1d-3bf6fef1a061\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" Nov 01 05:45:22.764173 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.763974 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7729d415-70e5-4be3-a9cc-bbd13e065c71-ca-trust-extracted\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:22.764173 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.763982 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-multus-cni-dir\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.764173 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.764006 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-var-lib-openvswitch\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.764173 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.764025 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrtw2\" (UniqueName: \"kubernetes.io/projected/ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8-kube-api-access-rrtw2\") pod \"node-resolver-2qtwx\" (UID: \"ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8\") " pod="openshift-dns/node-resolver-2qtwx" Nov 01 05:45:22.764173 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.764060 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxn5f\" (UniqueName: \"kubernetes.io/projected/de981773-b03b-4332-b2c1-b8d894423c57-kube-api-access-mxn5f\") pod \"service-ca-5fffc97bdf-zwlff\" (UID: \"de981773-b03b-4332-b2c1-b8d894423c57\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zwlff" Nov 01 05:45:22.764173 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.764078 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6cd67a78-09f1-4988-99b0-1ac9097f2f39-kubelet-dir\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:45:22.764173 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.764093 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5705a40-d3e4-40a8-a561-17c435f388b1-system-cni-dir\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:45:22.764173 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.764110 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-host-var-lib-cni-bin\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.764173 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.764133 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f98900-ff17-447f-8429-60c3e47fc7d1-tmp-dir\") pod \"dns-default-hxw5f\" (UID: \"72f98900-ff17-447f-8429-60c3e47fc7d1\") " pod="openshift-dns/dns-default-hxw5f" Nov 01 05:45:22.764173 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.764160 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8fz2\" (UniqueName: \"kubernetes.io/projected/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-kube-api-access-f8fz2\") pod \"network-metrics-daemon-tt8gq\" (UID: \"a1f5a77d-43d5-48bc-a8b5-5db8f1152853\") " pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:45:22.764618 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.764205 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdrd9\" (UniqueName: \"kubernetes.io/projected/0719dc90-dbae-425a-8439-812425e1b54c-kube-api-access-mdrd9\") pod \"iptables-alerter-qbg2s\" (UID: \"0719dc90-dbae-425a-8439-812425e1b54c\") " pod="openshift-network-operator/iptables-alerter-qbg2s" Nov 01 05:45:22.764618 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.764227 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5705a40-d3e4-40a8-a561-17c435f388b1-cni-binary-copy\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:45:22.764618 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.764234 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/580bebeb-9495-4541-8d1d-3bf6fef1a061-trusted-ca\") pod \"console-operator-7cd56b8dbc-njjn4\" (UID: \"580bebeb-9495-4541-8d1d-3bf6fef1a061\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" Nov 01 05:45:22.764618 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.764285 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-multus-socket-dir-parent\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.764618 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.764343 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f98900-ff17-447f-8429-60c3e47fc7d1-tmp-dir\") pod \"dns-default-hxw5f\" (UID: \"72f98900-ff17-447f-8429-60c3e47fc7d1\") " pod="openshift-dns/dns-default-hxw5f" Nov 01 05:45:22.764618 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.764468 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-host-var-lib-cni-multus\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.764618 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.764509 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbc9c127-21a0-4020-a093-d02b4061db5a-service-ca-bundle\") pod \"insights-operator-7c4986d449-r8n4b\" (UID: \"cbc9c127-21a0-4020-a093-d02b4061db5a\") " pod="openshift-insights/insights-operator-7c4986d449-r8n4b" Nov 01 05:45:22.764618 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.764528 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7dc7486d-3188-4bb8-b907-b653caa18660-dbus\") pod \"global-pull-secret-syncer-r5l82\" (UID: \"7dc7486d-3188-4bb8-b907-b653caa18660\") " pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:45:22.764618 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.764542 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-run-netns\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.764618 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.764608 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5705a40-d3e4-40a8-a561-17c435f388b1-cni-binary-copy\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:45:22.764967 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.764645 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-run-systemd\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.764967 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.764674 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbc9c127-21a0-4020-a093-d02b4061db5a-service-ca-bundle\") pod \"insights-operator-7c4986d449-r8n4b\" (UID: \"cbc9c127-21a0-4020-a093-d02b4061db5a\") " pod="openshift-insights/insights-operator-7c4986d449-r8n4b" Nov 01 05:45:22.764967 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.764677 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-trusted-ca-bundle\") pod \"periodic-gathering-nh7q2-cs6dz\" (UID: \"47088d4b-2aa6-4f69-9847-f361e4fa5f7a\") " pod="openshift-insights/periodic-gathering-nh7q2-cs6dz" Nov 01 05:45:22.764967 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.764696 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6r8g\" (UniqueName: \"kubernetes.io/projected/31894c07-ba1e-4865-9481-58555f919bcc-kube-api-access-t6r8g\") pod \"kas-connection-checker-685895c7c9-ls8v2\" (UID: \"31894c07-ba1e-4865-9481-58555f919bcc\") " pod="kube-system/kas-connection-checker-685895c7c9-ls8v2" Nov 01 05:45:22.764967 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.764713 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-run-ovn\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.764967 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.764731 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5rfj\" (UniqueName: \"kubernetes.io/projected/72f98900-ff17-447f-8429-60c3e47fc7d1-kube-api-access-r5rfj\") pod \"dns-default-hxw5f\" (UID: \"72f98900-ff17-447f-8429-60c3e47fc7d1\") " pod="openshift-dns/dns-default-hxw5f" Nov 01 05:45:22.764967 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.764775 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f5705a40-d3e4-40a8-a561-17c435f388b1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:45:22.764967 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.764813 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-host-var-lib-kubelet\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.764967 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.764860 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs\") pod \"network-metrics-daemon-tt8gq\" (UID: \"a1f5a77d-43d5-48bc-a8b5-5db8f1152853\") " pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:45:22.764967 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.764904 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/293a6ef2-f0c9-4390-ac82-028792a3b64d-serving-cert\") pod \"service-ca-operator-544d749c8c-vr978\" (UID: \"293a6ef2-f0c9-4390-ac82-028792a3b64d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vr978" Nov 01 05:45:22.764967 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.764911 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f5705a40-d3e4-40a8-a561-17c435f388b1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:45:22.764967 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.764926 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-trusted-ca-bundle\") pod \"periodic-gathering-nh7q2-cs6dz\" (UID: \"47088d4b-2aa6-4f69-9847-f361e4fa5f7a\") " pod="openshift-insights/periodic-gathering-nh7q2-cs6dz" Nov 01 05:45:22.764967 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.764962 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/197fb18c-6703-4deb-93b3-0186d7ceed67-konnectivity-ca\") pod \"konnectivity-agent-n7mks\" (UID: \"197fb18c-6703-4deb-93b3-0186d7ceed67\") " pod="kube-system/konnectivity-agent-n7mks" Nov 01 05:45:22.765656 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.765028 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.765656 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.765101 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cbc9c127-21a0-4020-a093-d02b4061db5a-snapshots\") pod \"insights-operator-7c4986d449-r8n4b\" (UID: \"cbc9c127-21a0-4020-a093-d02b4061db5a\") " pod="openshift-insights/insights-operator-7c4986d449-r8n4b" Nov 01 05:45:22.765656 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.765117 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/197fb18c-6703-4deb-93b3-0186d7ceed67-konnectivity-ca\") pod \"konnectivity-agent-n7mks\" (UID: \"197fb18c-6703-4deb-93b3-0186d7ceed67\") " pod="kube-system/konnectivity-agent-n7mks" Nov 01 05:45:22.765656 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.765147 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8-hosts-file\") pod \"node-resolver-2qtwx\" (UID: \"ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8\") " pod="openshift-dns/node-resolver-2qtwx" Nov 01 05:45:22.765656 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.765168 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/293a6ef2-f0c9-4390-ac82-028792a3b64d-serving-cert\") pod \"service-ca-operator-544d749c8c-vr978\" (UID: \"293a6ef2-f0c9-4390-ac82-028792a3b64d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vr978" Nov 01 05:45:22.765656 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.765173 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cbc9c127-21a0-4020-a093-d02b4061db5a-snapshots\") pod \"insights-operator-7c4986d449-r8n4b\" (UID: \"cbc9c127-21a0-4020-a093-d02b4061db5a\") " pod="openshift-insights/insights-operator-7c4986d449-r8n4b" Nov 01 05:45:22.765656 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.765207 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmjjx\" (UniqueName: \"kubernetes.io/projected/7db79928-e03b-4027-9cca-40191f6a300d-kube-api-access-jmjjx\") pod \"migrator-66df5d496b-wtlk8\" (UID: \"7db79928-e03b-4027-9cca-40191f6a300d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wtlk8" Nov 01 05:45:22.765656 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.765210 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-vbztt\"" Nov 01 05:45:22.765656 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.765225 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f9ccbc38-43f9-4396-bb59-2f746b3d639d-trusted-ca\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:22.765656 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.765242 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/72f98900-ff17-447f-8429-60c3e47fc7d1-config-volume\") pod \"dns-default-hxw5f\" (UID: \"72f98900-ff17-447f-8429-60c3e47fc7d1\") " pod="openshift-dns/dns-default-hxw5f" Nov 01 05:45:22.765656 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.765226 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 05:45:22.765656 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.765260 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-sys\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.765656 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.765288 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgrrf\" (UniqueName: \"kubernetes.io/projected/f4051a42-5120-4b67-b904-5db6b4897e07-kube-api-access-vgrrf\") pod \"volume-data-source-validator-58c9f5f46d-54lc7\" (UID: \"f4051a42-5120-4b67-b904-5db6b4897e07\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-54lc7" Nov 01 05:45:22.765656 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.765314 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7dc7486d-3188-4bb8-b907-b653caa18660-original-pull-secret\") pod \"global-pull-secret-syncer-r5l82\" (UID: \"7dc7486d-3188-4bb8-b907-b653caa18660\") " pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:45:22.765656 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.765446 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-m2d6d\"" Nov 01 05:45:22.765656 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.765509 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6cd67a78-09f1-4988-99b0-1ac9097f2f39-device-dir\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:45:22.765656 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.765543 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-bound-sa-token\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:22.765656 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.765587 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7dc7486d-3188-4bb8-b907-b653caa18660-original-pull-secret\") pod \"global-pull-secret-syncer-r5l82\" (UID: \"7dc7486d-3188-4bb8-b907-b653caa18660\") " pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:45:22.765656 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.765614 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swfdf\" (UniqueName: \"kubernetes.io/projected/5e70277c-fae5-4e5a-b3d0-743d7485d2b3-kube-api-access-swfdf\") pod \"kas-connection-checker-685895c7c9-b2659\" (UID: \"5e70277c-fae5-4e5a-b3d0-743d7485d2b3\") " pod="kube-system/kas-connection-checker-685895c7c9-b2659" Nov 01 05:45:22.765656 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.765647 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2191a9b0-b911-46e9-8ed4-b5119e37341d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-54tr8\" (UID: \"2191a9b0-b911-46e9-8ed4-b5119e37341d\") " pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:22.765656 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.765666 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2xnh\" (UniqueName: \"kubernetes.io/projected/529e2d97-b167-4011-a215-02ece8519c00-kube-api-access-k2xnh\") pod \"network-check-target-ljcjx\" (UID: \"529e2d97-b167-4011-a215-02ece8519c00\") " pod="openshift-network-diagnostics/network-check-target-ljcjx" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.765685 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-etc-modprobe-d\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.765711 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-host-run-netns\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.765718 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-f4d6z\"" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.765736 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gcvx\" (UniqueName: \"kubernetes.io/projected/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-kube-api-access-5gcvx\") pod \"periodic-gathering-nh7q2-cs6dz\" (UID: \"47088d4b-2aa6-4f69-9847-f361e4fa5f7a\") " pod="openshift-insights/periodic-gathering-nh7q2-cs6dz" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.765752 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.765763 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f9ccbc38-43f9-4396-bb59-2f746b3d639d-image-registry-private-configuration\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.765809 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-default-certificate\") pod \"router-default-6c5b5d5bd5-776bc\" (UID: \"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78\") " pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.765851 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqqbs\" (UniqueName: \"kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-kube-api-access-vqqbs\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.765879 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72f98900-ff17-447f-8429-60c3e47fc7d1-metrics-tls\") pod \"dns-default-hxw5f\" (UID: \"72f98900-ff17-447f-8429-60c3e47fc7d1\") " pod="openshift-dns/dns-default-hxw5f" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.765906 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-etc-sysctl-d\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.765933 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7729d415-70e5-4be3-a9cc-bbd13e065c71-trusted-ca\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.765950 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f9ccbc38-43f9-4396-bb59-2f746b3d639d-trusted-ca\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.765960 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-log-socket\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.765986 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/21519345-86b8-441e-a395-1e8385cd3732-ovn-node-metrics-cert\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.766004 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.766024 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzj6t\" (UniqueName: \"kubernetes.io/projected/1b19c3b0-887e-4e27-bac3-028003f1aac5-kube-api-access-fzj6t\") pod \"ingress-canary-dwklx\" (UID: \"1b19c3b0-887e-4e27-bac3-028003f1aac5\") " pod="openshift-ingress-canary/ingress-canary-dwklx" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.766059 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cbc9c127-21a0-4020-a093-d02b4061db5a-serving-cert\") pod \"insights-operator-7c4986d449-r8n4b\" (UID: \"cbc9c127-21a0-4020-a093-d02b4061db5a\") " pod="openshift-insights/insights-operator-7c4986d449-r8n4b" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.766089 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pdtj\" (UniqueName: \"kubernetes.io/projected/293a6ef2-f0c9-4390-ac82-028792a3b64d-kube-api-access-6pdtj\") pod \"service-ca-operator-544d749c8c-vr978\" (UID: \"293a6ef2-f0c9-4390-ac82-028792a3b64d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vr978" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.766099 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.766115 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/109327c0-f8f8-4dde-9cfe-489bf5a7eaad-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-95sds\" (UID: \"109327c0-f8f8-4dde-9cfe-489bf5a7eaad\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95sds" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.766119 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.766116 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-default-certificate\") pod \"router-default-6c5b5d5bd5-776bc\" (UID: \"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78\") " pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.766200 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.766199 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f9ccbc38-43f9-4396-bb59-2f746b3d639d-image-registry-private-configuration\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.766245 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.766296 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.766310 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.766333 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-bound-sa-token\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.766392 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7729d415-70e5-4be3-a9cc-bbd13e065c71-registry-certificates\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.766349 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.766314 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.766401 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.766511 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cbc9c127-21a0-4020-a093-d02b4061db5a-serving-cert\") pod \"insights-operator-7c4986d449-r8n4b\" (UID: \"cbc9c127-21a0-4020-a093-d02b4061db5a\") " pod="openshift-insights/insights-operator-7c4986d449-r8n4b" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.766523 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/109327c0-f8f8-4dde-9cfe-489bf5a7eaad-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-95sds\" (UID: \"109327c0-f8f8-4dde-9cfe-489bf5a7eaad\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95sds" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.766552 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rz4ht\"" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.766572 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6cd67a78-09f1-4988-99b0-1ac9097f2f39-registration-dir\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.766582 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/109327c0-f8f8-4dde-9cfe-489bf5a7eaad-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-95sds\" (UID: \"109327c0-f8f8-4dde-9cfe-489bf5a7eaad\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95sds" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.766648 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.766659 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7729d415-70e5-4be3-a9cc-bbd13e065c71-trusted-ca\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.766682 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/109327c0-f8f8-4dde-9cfe-489bf5a7eaad-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-95sds\" (UID: \"109327c0-f8f8-4dde-9cfe-489bf5a7eaad\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95sds" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.766717 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/21519345-86b8-441e-a395-1e8385cd3732-ovn-node-metrics-cert\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.766729 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/375debb4-cfe1-44ab-907b-0a34a031c2ea-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7grdp\" (UID: \"375debb4-cfe1-44ab-907b-0a34a031c2ea\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7grdp" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.766735 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7729d415-70e5-4be3-a9cc-bbd13e065c71-registry-certificates\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.766746 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9tjqf\"" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.766768 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-cni-bin\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.766802 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/21519345-86b8-441e-a395-1e8385cd3732-ovnkube-config\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.766819 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rpsgx\"" Nov 01 05:45:22.766814 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.766839 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/21519345-86b8-441e-a395-1e8385cd3732-env-overrides\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.766892 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.766967 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5zklw\"" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.766965 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-etc-sysctl-conf\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.767008 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/21519345-86b8-441e-a395-1e8385cd3732-ovnkube-config\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.767029 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.767035 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/197fb18c-6703-4deb-93b3-0186d7ceed67-agent-certs\") pod \"konnectivity-agent-n7mks\" (UID: \"197fb18c-6703-4deb-93b3-0186d7ceed67\") " pod="kube-system/konnectivity-agent-n7mks" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.767070 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/21519345-86b8-441e-a395-1e8385cd3732-env-overrides\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.767073 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dc9422c2-b041-4f6b-8495-961a9571c1e4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-9j8j4\" (UID: \"dc9422c2-b041-4f6b-8495-961a9571c1e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9j8j4" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.767113 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s56k\" (UniqueName: \"kubernetes.io/projected/2acbc647-2361-4935-b503-5bd9af74f822-kube-api-access-4s56k\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.767114 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-gvvtc\"" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.767129 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7729d415-70e5-4be3-a9cc-bbd13e065c71-image-registry-private-configuration\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.767149 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-registry-tls\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.767171 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72qwb\" (UniqueName: \"kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-kube-api-access-72qwb\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.767191 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5705a40-d3e4-40a8-a561-17c435f388b1-cnibin\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.767195 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-lgfkm\"" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.767206 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzxml\" (UniqueName: \"kubernetes.io/projected/f5705a40-d3e4-40a8-a561-17c435f388b1-kube-api-access-dzxml\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.767234 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hksk\" (UniqueName: \"kubernetes.io/projected/cbc9c127-21a0-4020-a093-d02b4061db5a-kube-api-access-6hksk\") pod \"insights-operator-7c4986d449-r8n4b\" (UID: \"cbc9c127-21a0-4020-a093-d02b4061db5a\") " pod="openshift-insights/insights-operator-7c4986d449-r8n4b" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.767274 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.767279 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-etc-kubernetes\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.767306 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.767343 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7dc7486d-3188-4bb8-b907-b653caa18660-kubelet-config\") pod \"global-pull-secret-syncer-r5l82\" (UID: \"7dc7486d-3188-4bb8-b907-b653caa18660\") " pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.767366 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/197fb18c-6703-4deb-93b3-0186d7ceed67-agent-certs\") pod \"konnectivity-agent-n7mks\" (UID: \"197fb18c-6703-4deb-93b3-0186d7ceed67\") " pod="kube-system/konnectivity-agent-n7mks" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.767381 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-74ptq\"" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.767406 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dbh2\" (UniqueName: \"kubernetes.io/projected/cde21439-a494-47cd-8cc5-161fc96a6701-kube-api-access-8dbh2\") pod \"node-ca-tw96g\" (UID: \"cde21439-a494-47cd-8cc5-161fc96a6701\") " pod="openshift-image-registry/node-ca-tw96g" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.767426 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7729d415-70e5-4be3-a9cc-bbd13e065c71-image-registry-private-configuration\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.767801 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.767814 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.767919 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-bswc2\"" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.767989 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-node-log\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.768010 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-vp9l2\"" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.768032 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lmfd\" (UniqueName: \"kubernetes.io/projected/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-kube-api-access-2lmfd\") pod \"cluster-samples-operator-6ddb6c8b94-flt78\" (UID: \"320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flt78" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.768269 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.768340 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qfzvw\"" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.768360 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/375debb4-cfe1-44ab-907b-0a34a031c2ea-nginx-conf\") pod \"networking-console-plugin-86c478bb88-7grdp\" (UID: \"375debb4-cfe1-44ab-907b-0a34a031c2ea\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7grdp" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.768407 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6cd67a78-09f1-4988-99b0-1ac9097f2f39-etc-selinux\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.768414 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/375debb4-cfe1-44ab-907b-0a34a031c2ea-nginx-conf\") pod \"networking-console-plugin-86c478bb88-7grdp\" (UID: \"375debb4-cfe1-44ab-907b-0a34a031c2ea\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7grdp" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.768448 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbc9c127-21a0-4020-a093-d02b4061db5a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-r8n4b\" (UID: \"cbc9c127-21a0-4020-a093-d02b4061db5a\") " pod="openshift-insights/insights-operator-7c4986d449-r8n4b" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.768562 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-multus-conf-dir\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.768621 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4v9k\" (UniqueName: \"kubernetes.io/projected/2191a9b0-b911-46e9-8ed4-b5119e37341d-kube-api-access-b4v9k\") pod \"insights-runtime-extractor-54tr8\" (UID: \"2191a9b0-b911-46e9-8ed4-b5119e37341d\") " pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.768663 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f9ccbc38-43f9-4396-bb59-2f746b3d639d-installation-pull-secrets\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.768699 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-etc-systemd\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.768815 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f5705a40-d3e4-40a8-a561-17c435f388b1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.768847 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-etc-kubernetes\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.768883 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdc27\" (UniqueName: \"kubernetes.io/projected/f8238f6d-c793-4f76-b63b-d2d50f9a585d-kube-api-access-cdc27\") pod \"network-check-source-6f78d59f99-pgxkq\" (UID: \"f8238f6d-c793-4f76-b63b-d2d50f9a585d\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-pgxkq" Nov 01 05:45:22.768907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.768916 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b19c3b0-887e-4e27-bac3-028003f1aac5-cert\") pod \"ingress-canary-dwklx\" (UID: \"1b19c3b0-887e-4e27-bac3-028003f1aac5\") " pod="openshift-ingress-canary/ingress-canary-dwklx" Nov 01 05:45:22.770887 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.768965 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk8zm\" (UniqueName: \"kubernetes.io/projected/580bebeb-9495-4541-8d1d-3bf6fef1a061-kube-api-access-mk8zm\") pod \"console-operator-7cd56b8dbc-njjn4\" (UID: \"580bebeb-9495-4541-8d1d-3bf6fef1a061\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" Nov 01 05:45:22.770887 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.768999 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5705a40-d3e4-40a8-a561-17c435f388b1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:45:22.770887 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.769031 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67ae987f-79b9-4852-98bd-e0537c412b17-cni-binary-copy\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.770887 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.769079 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-kubelet\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.770887 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.769173 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqxsx\" (UniqueName: \"kubernetes.io/projected/baad0c0a-7a5d-454d-8ea9-4a3b27115992-kube-api-access-kqxsx\") pod \"migrator-66df5d496b-n6nns\" (UID: \"baad0c0a-7a5d-454d-8ea9-4a3b27115992\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n6nns" Nov 01 05:45:22.770887 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.769202 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2acbc647-2361-4935-b503-5bd9af74f822-etc-tuned\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.770887 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.769234 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/580bebeb-9495-4541-8d1d-3bf6fef1a061-config\") pod \"console-operator-7cd56b8dbc-njjn4\" (UID: \"580bebeb-9495-4541-8d1d-3bf6fef1a061\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" Nov 01 05:45:22.770887 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.769265 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2acbc647-2361-4935-b503-5bd9af74f822-tmp\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.770887 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.769295 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-host-run-k8s-cni-cncf-io\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.770887 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.769375 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsp9k\" (UniqueName: \"kubernetes.io/projected/dc9422c2-b041-4f6b-8495-961a9571c1e4-kube-api-access-jsp9k\") pod \"cluster-monitoring-operator-884576c57-9j8j4\" (UID: \"dc9422c2-b041-4f6b-8495-961a9571c1e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9j8j4" Nov 01 05:45:22.770887 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.769404 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-run-openvswitch\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.770887 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.769438 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-cni-netd\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.770887 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.769498 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6xv7\" (UniqueName: \"kubernetes.io/projected/21519345-86b8-441e-a395-1e8385cd3732-kube-api-access-g6xv7\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.770887 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.769581 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f9ccbc38-43f9-4396-bb59-2f746b3d639d-installation-pull-secrets\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:22.770887 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.769594 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 05:45:22.770887 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.769601 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/580bebeb-9495-4541-8d1d-3bf6fef1a061-serving-cert\") pod \"console-operator-7cd56b8dbc-njjn4\" (UID: \"580bebeb-9495-4541-8d1d-3bf6fef1a061\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" Nov 01 05:45:22.770887 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.769656 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2191a9b0-b911-46e9-8ed4-b5119e37341d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54tr8\" (UID: \"2191a9b0-b911-46e9-8ed4-b5119e37341d\") " pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:22.770887 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.769693 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/293a6ef2-f0c9-4390-ac82-028792a3b64d-config\") pod \"service-ca-operator-544d749c8c-vr978\" (UID: \"293a6ef2-f0c9-4390-ac82-028792a3b64d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vr978" Nov 01 05:45:22.770887 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.769724 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz6tw\" (UniqueName: \"kubernetes.io/projected/6cd67a78-09f1-4988-99b0-1ac9097f2f39-kube-api-access-kz6tw\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:45:22.770887 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.769759 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gbgq\" (UniqueName: \"kubernetes.io/projected/67ae987f-79b9-4852-98bd-e0537c412b17-kube-api-access-7gbgq\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.770887 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.769851 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/580bebeb-9495-4541-8d1d-3bf6fef1a061-serving-cert\") pod \"console-operator-7cd56b8dbc-njjn4\" (UID: \"580bebeb-9495-4541-8d1d-3bf6fef1a061\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" Nov 01 05:45:22.770887 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.769874 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2nbg\" (UniqueName: \"kubernetes.io/projected/1cc9d6f4-202c-45bb-8a4c-4b8e936874de-kube-api-access-n2nbg\") pod \"kas-connection-checker-685895c7c9-b2kmh\" (UID: \"1cc9d6f4-202c-45bb-8a4c-4b8e936874de\") " pod="kube-system/kas-connection-checker-685895c7c9-b2kmh" Nov 01 05:45:22.770887 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.769910 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc9422c2-b041-4f6b-8495-961a9571c1e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9j8j4\" (UID: \"dc9422c2-b041-4f6b-8495-961a9571c1e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9j8j4" Nov 01 05:45:22.770887 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.769944 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-run-ovn-kubernetes\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.770887 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.769972 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2191a9b0-b911-46e9-8ed4-b5119e37341d-crio-socket\") pod \"insights-runtime-extractor-54tr8\" (UID: \"2191a9b0-b911-46e9-8ed4-b5119e37341d\") " pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:22.770887 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.770047 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-flt78\" (UID: \"320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flt78" Nov 01 05:45:22.770887 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.770078 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0719dc90-dbae-425a-8439-812425e1b54c-host-slash\") pod \"iptables-alerter-qbg2s\" (UID: \"0719dc90-dbae-425a-8439-812425e1b54c\") " pod="openshift-network-operator/iptables-alerter-qbg2s" Nov 01 05:45:22.770887 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.770110 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6cd67a78-09f1-4988-99b0-1ac9097f2f39-socket-dir\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:45:22.770887 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.770156 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-etc-openvswitch\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.770887 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.770158 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbc9c127-21a0-4020-a093-d02b4061db5a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-r8n4b\" (UID: \"cbc9c127-21a0-4020-a093-d02b4061db5a\") " pod="openshift-insights/insights-operator-7c4986d449-r8n4b" Nov 01 05:45:22.770887 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.770894 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbc9c127-21a0-4020-a093-d02b4061db5a-tmp\") pod \"insights-operator-7c4986d449-r8n4b\" (UID: \"cbc9c127-21a0-4020-a093-d02b4061db5a\") " pod="openshift-insights/insights-operator-7c4986d449-r8n4b" Nov 01 05:45:22.772271 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.770932 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-flt78\" (UID: \"320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flt78" Nov 01 05:45:22.772271 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.770978 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/de981773-b03b-4332-b2c1-b8d894423c57-signing-cabundle\") pod \"service-ca-5fffc97bdf-zwlff\" (UID: \"de981773-b03b-4332-b2c1-b8d894423c57\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zwlff" Nov 01 05:45:22.772271 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.771024 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-registry-tls\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:22.772271 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.771058 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f9ccbc38-43f9-4396-bb59-2f746b3d639d-ca-trust-extracted\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:22.772271 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.771087 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-etc-sysconfig\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.772271 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.771117 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6cd67a78-09f1-4988-99b0-1ac9097f2f39-sys-fs\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:45:22.772271 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.771156 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-hostroot\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.772271 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.771161 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 01 05:45:22.772271 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.771182 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f5705a40-d3e4-40a8-a561-17c435f388b1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:45:22.772271 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.771189 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8-tmp-dir\") pod \"node-resolver-2qtwx\" (UID: \"ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8\") " pod="openshift-dns/node-resolver-2qtwx" Nov 01 05:45:22.772271 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.771223 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f9ccbc38-43f9-4396-bb59-2f746b3d639d-registry-certificates\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:22.772271 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.771260 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-service-ca-bundle\") pod \"router-default-6c5b5d5bd5-776bc\" (UID: \"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78\") " pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:22.772271 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.771268 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f9ccbc38-43f9-4396-bb59-2f746b3d639d-ca-trust-extracted\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:22.772271 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.771297 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7729d415-70e5-4be3-a9cc-bbd13e065c71-installation-pull-secrets\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:22.772271 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.771369 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/580bebeb-9495-4541-8d1d-3bf6fef1a061-config\") pod \"console-operator-7cd56b8dbc-njjn4\" (UID: \"580bebeb-9495-4541-8d1d-3bf6fef1a061\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" Nov 01 05:45:22.772271 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.771530 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b19c3b0-887e-4e27-bac3-028003f1aac5-cert\") pod \"ingress-canary-dwklx\" (UID: \"1b19c3b0-887e-4e27-bac3-028003f1aac5\") " pod="openshift-ingress-canary/ingress-canary-dwklx" Nov 01 05:45:22.772271 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.771907 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67ae987f-79b9-4852-98bd-e0537c412b17-cni-binary-copy\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.772271 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.771992 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/293a6ef2-f0c9-4390-ac82-028792a3b64d-config\") pod \"service-ca-operator-544d749c8c-vr978\" (UID: \"293a6ef2-f0c9-4390-ac82-028792a3b64d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vr978" Nov 01 05:45:22.772271 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.772085 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f9ccbc38-43f9-4396-bb59-2f746b3d639d-registry-certificates\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:22.772271 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.772092 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbc9c127-21a0-4020-a093-d02b4061db5a-tmp\") pod \"insights-operator-7c4986d449-r8n4b\" (UID: \"cbc9c127-21a0-4020-a093-d02b4061db5a\") " pod="openshift-insights/insights-operator-7c4986d449-r8n4b" Nov 01 05:45:22.772271 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.772087 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-flt78\" (UID: \"320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flt78" Nov 01 05:45:22.772271 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.772146 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8-tmp-dir\") pod \"node-resolver-2qtwx\" (UID: \"ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8\") " pod="openshift-dns/node-resolver-2qtwx" Nov 01 05:45:22.772271 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.772164 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2acbc647-2361-4935-b503-5bd9af74f822-tmp\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.772271 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.772205 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2acbc647-2361-4935-b503-5bd9af74f822-etc-tuned\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.772271 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.772206 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/de981773-b03b-4332-b2c1-b8d894423c57-signing-cabundle\") pod \"service-ca-5fffc97bdf-zwlff\" (UID: \"de981773-b03b-4332-b2c1-b8d894423c57\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zwlff" Nov 01 05:45:22.773308 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.772397 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7729d415-70e5-4be3-a9cc-bbd13e065c71-installation-pull-secrets\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:22.773308 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.773237 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/de981773-b03b-4332-b2c1-b8d894423c57-signing-key\") pod \"service-ca-5fffc97bdf-zwlff\" (UID: \"de981773-b03b-4332-b2c1-b8d894423c57\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zwlff" Nov 01 05:45:22.773308 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.773255 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/67ae987f-79b9-4852-98bd-e0537c412b17-multus-daemon-config\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.773446 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.773437 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/21519345-86b8-441e-a395-1e8385cd3732-ovnkube-script-lib\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.773796 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.773774 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0719dc90-dbae-425a-8439-812425e1b54c-iptables-alerter-script\") pod \"iptables-alerter-qbg2s\" (UID: \"0719dc90-dbae-425a-8439-812425e1b54c\") " pod="openshift-network-operator/iptables-alerter-qbg2s" Nov 01 05:45:22.776028 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.776009 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2191a9b0-b911-46e9-8ed4-b5119e37341d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-54tr8\" (UID: \"2191a9b0-b911-46e9-8ed4-b5119e37341d\") " pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:22.776114 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.776102 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/72f98900-ff17-447f-8429-60c3e47fc7d1-config-volume\") pod \"dns-default-hxw5f\" (UID: \"72f98900-ff17-447f-8429-60c3e47fc7d1\") " pod="openshift-dns/dns-default-hxw5f" Nov 01 05:45:22.777449 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.777434 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dc9422c2-b041-4f6b-8495-961a9571c1e4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-9j8j4\" (UID: \"dc9422c2-b041-4f6b-8495-961a9571c1e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9j8j4" Nov 01 05:45:22.826949 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.826916 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j64kh\" (UniqueName: \"kubernetes.io/projected/109327c0-f8f8-4dde-9cfe-489bf5a7eaad-kube-api-access-j64kh\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-95sds\" (UID: \"109327c0-f8f8-4dde-9cfe-489bf5a7eaad\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95sds" Nov 01 05:45:22.827083 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.826977 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5rfj\" (UniqueName: \"kubernetes.io/projected/72f98900-ff17-447f-8429-60c3e47fc7d1-kube-api-access-r5rfj\") pod \"dns-default-hxw5f\" (UID: \"72f98900-ff17-447f-8429-60c3e47fc7d1\") " pod="openshift-dns/dns-default-hxw5f" Nov 01 05:45:22.827083 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.827026 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgrrf\" (UniqueName: \"kubernetes.io/projected/f4051a42-5120-4b67-b904-5db6b4897e07-kube-api-access-vgrrf\") pod \"volume-data-source-validator-58c9f5f46d-54lc7\" (UID: \"f4051a42-5120-4b67-b904-5db6b4897e07\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-54lc7" Nov 01 05:45:22.827157 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.827123 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8fz2\" (UniqueName: \"kubernetes.io/projected/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-kube-api-access-f8fz2\") pod \"network-metrics-daemon-tt8gq\" (UID: \"a1f5a77d-43d5-48bc-a8b5-5db8f1152853\") " pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:45:22.827157 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.827136 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7854\" (UniqueName: \"kubernetes.io/projected/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-kube-api-access-b7854\") pod \"router-default-6c5b5d5bd5-776bc\" (UID: \"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78\") " pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:22.827218 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.827138 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxn5f\" (UniqueName: \"kubernetes.io/projected/de981773-b03b-4332-b2c1-b8d894423c57-kube-api-access-mxn5f\") pod \"service-ca-5fffc97bdf-zwlff\" (UID: \"de981773-b03b-4332-b2c1-b8d894423c57\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zwlff" Nov 01 05:45:22.831929 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.831762 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swfdf\" (UniqueName: \"kubernetes.io/projected/5e70277c-fae5-4e5a-b3d0-743d7485d2b3-kube-api-access-swfdf\") pod \"kas-connection-checker-685895c7c9-b2659\" (UID: \"5e70277c-fae5-4e5a-b3d0-743d7485d2b3\") " pod="kube-system/kas-connection-checker-685895c7c9-b2659" Nov 01 05:45:22.831929 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.831796 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6r8g\" (UniqueName: \"kubernetes.io/projected/31894c07-ba1e-4865-9481-58555f919bcc-kube-api-access-t6r8g\") pod \"kas-connection-checker-685895c7c9-ls8v2\" (UID: \"31894c07-ba1e-4865-9481-58555f919bcc\") " pod="kube-system/kas-connection-checker-685895c7c9-ls8v2" Nov 01 05:45:22.831929 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.831821 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmjjx\" (UniqueName: \"kubernetes.io/projected/7db79928-e03b-4027-9cca-40191f6a300d-kube-api-access-jmjjx\") pod \"migrator-66df5d496b-wtlk8\" (UID: \"7db79928-e03b-4027-9cca-40191f6a300d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wtlk8" Nov 01 05:45:22.831929 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.831883 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2xnh\" (UniqueName: \"kubernetes.io/projected/529e2d97-b167-4011-a215-02ece8519c00-kube-api-access-k2xnh\") pod \"network-check-target-ljcjx\" (UID: \"529e2d97-b167-4011-a215-02ece8519c00\") " pod="openshift-network-diagnostics/network-check-target-ljcjx" Nov 01 05:45:22.831929 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.831917 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-bound-sa-token\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:22.832166 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.831961 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzj6t\" (UniqueName: \"kubernetes.io/projected/1b19c3b0-887e-4e27-bac3-028003f1aac5-kube-api-access-fzj6t\") pod \"ingress-canary-dwklx\" (UID: \"1b19c3b0-887e-4e27-bac3-028003f1aac5\") " pod="openshift-ingress-canary/ingress-canary-dwklx" Nov 01 05:45:22.832166 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.831964 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s56k\" (UniqueName: \"kubernetes.io/projected/2acbc647-2361-4935-b503-5bd9af74f822-kube-api-access-4s56k\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.832270 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.832241 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrtw2\" (UniqueName: \"kubernetes.io/projected/ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8-kube-api-access-rrtw2\") pod \"node-resolver-2qtwx\" (UID: \"ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8\") " pod="openshift-dns/node-resolver-2qtwx" Nov 01 05:45:22.832672 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.832657 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqqbs\" (UniqueName: \"kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-kube-api-access-vqqbs\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:22.834322 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.834305 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gcvx\" (UniqueName: \"kubernetes.io/projected/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-kube-api-access-5gcvx\") pod \"periodic-gathering-nh7q2-cs6dz\" (UID: \"47088d4b-2aa6-4f69-9847-f361e4fa5f7a\") " pod="openshift-insights/periodic-gathering-nh7q2-cs6dz" Nov 01 05:45:22.835424 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.835407 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdrd9\" (UniqueName: \"kubernetes.io/projected/0719dc90-dbae-425a-8439-812425e1b54c-kube-api-access-mdrd9\") pod \"iptables-alerter-qbg2s\" (UID: \"0719dc90-dbae-425a-8439-812425e1b54c\") " pod="openshift-network-operator/iptables-alerter-qbg2s" Nov 01 05:45:22.835528 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.835513 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pdtj\" (UniqueName: \"kubernetes.io/projected/293a6ef2-f0c9-4390-ac82-028792a3b64d-kube-api-access-6pdtj\") pod \"service-ca-operator-544d749c8c-vr978\" (UID: \"293a6ef2-f0c9-4390-ac82-028792a3b64d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vr978" Nov 01 05:45:22.836108 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.836092 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-bound-sa-token\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:22.839596 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.839546 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4v9k\" (UniqueName: \"kubernetes.io/projected/2191a9b0-b911-46e9-8ed4-b5119e37341d-kube-api-access-b4v9k\") pod \"insights-runtime-extractor-54tr8\" (UID: \"2191a9b0-b911-46e9-8ed4-b5119e37341d\") " pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:22.839702 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.839614 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz6tw\" (UniqueName: \"kubernetes.io/projected/6cd67a78-09f1-4988-99b0-1ac9097f2f39-kube-api-access-kz6tw\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:45:22.839702 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.839632 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk8zm\" (UniqueName: \"kubernetes.io/projected/580bebeb-9495-4541-8d1d-3bf6fef1a061-kube-api-access-mk8zm\") pod \"console-operator-7cd56b8dbc-njjn4\" (UID: \"580bebeb-9495-4541-8d1d-3bf6fef1a061\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" Nov 01 05:45:22.839849 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.839751 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lmfd\" (UniqueName: \"kubernetes.io/projected/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-kube-api-access-2lmfd\") pod \"cluster-samples-operator-6ddb6c8b94-flt78\" (UID: \"320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flt78" Nov 01 05:45:22.839849 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.839782 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdc27\" (UniqueName: \"kubernetes.io/projected/f8238f6d-c793-4f76-b63b-d2d50f9a585d-kube-api-access-cdc27\") pod \"network-check-source-6f78d59f99-pgxkq\" (UID: \"f8238f6d-c793-4f76-b63b-d2d50f9a585d\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-pgxkq" Nov 01 05:45:22.839849 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.839825 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dbh2\" (UniqueName: \"kubernetes.io/projected/cde21439-a494-47cd-8cc5-161fc96a6701-kube-api-access-8dbh2\") pod \"node-ca-tw96g\" (UID: \"cde21439-a494-47cd-8cc5-161fc96a6701\") " pod="openshift-image-registry/node-ca-tw96g" Nov 01 05:45:22.840008 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.839860 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72qwb\" (UniqueName: \"kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-kube-api-access-72qwb\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:22.840008 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.839893 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsp9k\" (UniqueName: \"kubernetes.io/projected/dc9422c2-b041-4f6b-8495-961a9571c1e4-kube-api-access-jsp9k\") pod \"cluster-monitoring-operator-884576c57-9j8j4\" (UID: \"dc9422c2-b041-4f6b-8495-961a9571c1e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9j8j4" Nov 01 05:45:22.840806 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.840733 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzxml\" (UniqueName: \"kubernetes.io/projected/f5705a40-d3e4-40a8-a561-17c435f388b1-kube-api-access-dzxml\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:45:22.840968 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.840945 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqxsx\" (UniqueName: \"kubernetes.io/projected/baad0c0a-7a5d-454d-8ea9-4a3b27115992-kube-api-access-kqxsx\") pod \"migrator-66df5d496b-n6nns\" (UID: \"baad0c0a-7a5d-454d-8ea9-4a3b27115992\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n6nns" Nov 01 05:45:22.841089 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.841074 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hksk\" (UniqueName: \"kubernetes.io/projected/cbc9c127-21a0-4020-a093-d02b4061db5a-kube-api-access-6hksk\") pod \"insights-operator-7c4986d449-r8n4b\" (UID: \"cbc9c127-21a0-4020-a093-d02b4061db5a\") " pod="openshift-insights/insights-operator-7c4986d449-r8n4b" Nov 01 05:45:22.841355 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.841337 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2nbg\" (UniqueName: \"kubernetes.io/projected/1cc9d6f4-202c-45bb-8a4c-4b8e936874de-kube-api-access-n2nbg\") pod \"kas-connection-checker-685895c7c9-b2kmh\" (UID: \"1cc9d6f4-202c-45bb-8a4c-4b8e936874de\") " pod="kube-system/kas-connection-checker-685895c7c9-b2kmh" Nov 01 05:45:22.844201 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.844182 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gbgq\" (UniqueName: \"kubernetes.io/projected/67ae987f-79b9-4852-98bd-e0537c412b17-kube-api-access-7gbgq\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.844647 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.844626 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6xv7\" (UniqueName: \"kubernetes.io/projected/21519345-86b8-441e-a395-1e8385cd3732-kube-api-access-g6xv7\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.872487 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.872455 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-multus-conf-dir\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.872643 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.872511 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-etc-systemd\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.872643 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.872567 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-multus-conf-dir\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.872643 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.872581 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-etc-systemd\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.872643 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.872607 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-etc-kubernetes\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.872862 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.872629 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5705a40-d3e4-40a8-a561-17c435f388b1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:45:22.872862 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.872663 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-kubelet\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.872862 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.872670 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-etc-kubernetes\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.872862 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.872686 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-host-run-k8s-cni-cncf-io\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.872862 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.872713 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-host-run-k8s-cni-cncf-io\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.872862 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.872714 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5705a40-d3e4-40a8-a561-17c435f388b1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:45:22.872862 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.872729 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-run-openvswitch\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.872862 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.872753 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-cni-netd\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.872862 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.872762 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-run-openvswitch\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.872862 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.872770 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2191a9b0-b911-46e9-8ed4-b5119e37341d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54tr8\" (UID: \"2191a9b0-b911-46e9-8ed4-b5119e37341d\") " pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:22.872862 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.872789 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc9422c2-b041-4f6b-8495-961a9571c1e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9j8j4\" (UID: \"dc9422c2-b041-4f6b-8495-961a9571c1e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9j8j4" Nov 01 05:45:22.872862 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.872814 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-run-ovn-kubernetes\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.872862 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.872817 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-cni-netd\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.872862 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.872728 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-kubelet\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.872862 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.872835 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2191a9b0-b911-46e9-8ed4-b5119e37341d-crio-socket\") pod \"insights-runtime-extractor-54tr8\" (UID: \"2191a9b0-b911-46e9-8ed4-b5119e37341d\") " pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:22.872862 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.872857 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0719dc90-dbae-425a-8439-812425e1b54c-host-slash\") pod \"iptables-alerter-qbg2s\" (UID: \"0719dc90-dbae-425a-8439-812425e1b54c\") " pod="openshift-network-operator/iptables-alerter-qbg2s" Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.872859 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-run-ovn-kubernetes\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.872881 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2191a9b0-b911-46e9-8ed4-b5119e37341d-crio-socket\") pod \"insights-runtime-extractor-54tr8\" (UID: \"2191a9b0-b911-46e9-8ed4-b5119e37341d\") " pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.872879 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6cd67a78-09f1-4988-99b0-1ac9097f2f39-socket-dir\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.872912 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0719dc90-dbae-425a-8439-812425e1b54c-host-slash\") pod \"iptables-alerter-qbg2s\" (UID: \"0719dc90-dbae-425a-8439-812425e1b54c\") " pod="openshift-network-operator/iptables-alerter-qbg2s" Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.872929 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-etc-openvswitch\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:22.872901 5040 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.872956 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-flt78\" (UID: \"320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flt78" Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.872965 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6cd67a78-09f1-4988-99b0-1ac9097f2f39-socket-dir\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.872984 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-registry-tls\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873000 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-etc-openvswitch\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873012 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-etc-sysconfig\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:22.872902 5040 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:22.873041 5040 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873062 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-etc-sysconfig\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:22.873045 5040 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2191a9b0-b911-46e9-8ed4-b5119e37341d-insights-runtime-extractor-tls podName:2191a9b0-b911-46e9-8ed4-b5119e37341d nodeName:}" failed. No retries permitted until 2026-11-01 05:45:23.373023614 +0000 UTC m=+2.028140762 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2191a9b0-b911-46e9-8ed4-b5119e37341d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-54tr8" (UID: "2191a9b0-b911-46e9-8ed4-b5119e37341d") : secret "insights-runtime-extractor-tls" not found Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873098 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6cd67a78-09f1-4988-99b0-1ac9097f2f39-sys-fs\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873122 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-hostroot\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873127 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6cd67a78-09f1-4988-99b0-1ac9097f2f39-sys-fs\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:22.873160 5040 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc9422c2-b041-4f6b-8495-961a9571c1e4-cluster-monitoring-operator-tls podName:dc9422c2-b041-4f6b-8495-961a9571c1e4 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:23.373144359 +0000 UTC m=+2.028261517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dc9422c2-b041-4f6b-8495-961a9571c1e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9j8j4" (UID: "dc9422c2-b041-4f6b-8495-961a9571c1e4") : secret "cluster-monitoring-operator-tls" not found Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873150 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-service-ca-bundle\") pod \"router-default-6c5b5d5bd5-776bc\" (UID: \"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78\") " pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:22.873182 5040 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-samples-operator-tls podName:320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:23.373170895 +0000 UTC m=+2.028288040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-flt78" (UID: "320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2") : secret "samples-operator-tls" not found Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873211 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-slash\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873236 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-host\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873263 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-system-cni-dir\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873287 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cde21439-a494-47cd-8cc5-161fc96a6701-host\") pod \"node-ca-tw96g\" (UID: \"cde21439-a494-47cd-8cc5-161fc96a6701\") " pod="openshift-image-registry/node-ca-tw96g" Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873307 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-var-lib-kubelet\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873181 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-hostroot\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873329 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-run\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873330 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-slash\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873336 5040 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873377 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cde21439-a494-47cd-8cc5-161fc96a6701-host\") pod \"node-ca-tw96g\" (UID: \"cde21439-a494-47cd-8cc5-161fc96a6701\") " pod="openshift-image-registry/node-ca-tw96g" Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873390 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-host\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873415 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-host-run-multus-certs\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873372 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-host-run-multus-certs\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873436 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-system-cni-dir\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873440 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-run\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873450 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-systemd-units\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873474 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-var-lib-kubelet\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873498 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-systemd-units\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.873527 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873505 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-cnibin\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873587 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-cnibin\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873602 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-lib-modules\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873626 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-metrics-certs\") pod \"router-default-6c5b5d5bd5-776bc\" (UID: \"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78\") " pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873652 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-multus-cni-dir\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873668 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-var-lib-openvswitch\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873687 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6cd67a78-09f1-4988-99b0-1ac9097f2f39-kubelet-dir\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873715 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5705a40-d3e4-40a8-a561-17c435f388b1-system-cni-dir\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:22.873732 5040 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873742 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-host-var-lib-cni-bin\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873745 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-var-lib-openvswitch\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873765 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-lib-modules\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873778 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6cd67a78-09f1-4988-99b0-1ac9097f2f39-kubelet-dir\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873792 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5705a40-d3e4-40a8-a561-17c435f388b1-system-cni-dir\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:22.873796 5040 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-metrics-certs podName:fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:23.373781218 +0000 UTC m=+2.028898363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-metrics-certs") pod "router-default-6c5b5d5bd5-776bc" (UID: "fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78") : secret "router-metrics-certs-default" not found Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873734 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-multus-cni-dir\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873806 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-host-var-lib-cni-bin\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873826 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-multus-socket-dir-parent\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873857 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-host-var-lib-cni-multus\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873880 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7dc7486d-3188-4bb8-b907-b653caa18660-dbus\") pod \"global-pull-secret-syncer-r5l82\" (UID: \"7dc7486d-3188-4bb8-b907-b653caa18660\") " pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873894 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-multus-socket-dir-parent\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873901 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-service-ca-bundle\") pod \"router-default-6c5b5d5bd5-776bc\" (UID: \"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78\") " pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873904 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-run-netns\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873929 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-run-systemd\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873935 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-host-var-lib-cni-multus\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873970 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-run-netns\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.873977 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7dc7486d-3188-4bb8-b907-b653caa18660-dbus\") pod \"global-pull-secret-syncer-r5l82\" (UID: \"7dc7486d-3188-4bb8-b907-b653caa18660\") " pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874020 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-run-systemd\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874056 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-run-ovn\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874074 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-host-var-lib-kubelet\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874088 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs\") pod \"network-metrics-daemon-tt8gq\" (UID: \"a1f5a77d-43d5-48bc-a8b5-5db8f1152853\") " pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874105 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-run-ovn\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874105 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874130 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874144 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-host-var-lib-kubelet\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874157 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8-hosts-file\") pod \"node-resolver-2qtwx\" (UID: \"ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8\") " pod="openshift-dns/node-resolver-2qtwx" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874172 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8-hosts-file\") pod \"node-resolver-2qtwx\" (UID: \"ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8\") " pod="openshift-dns/node-resolver-2qtwx" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874185 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-sys\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:22.874204 5040 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874210 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6cd67a78-09f1-4988-99b0-1ac9097f2f39-device-dir\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874240 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-etc-modprobe-d\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874245 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-sys\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:22.874252 5040 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs podName:a1f5a77d-43d5-48bc-a8b5-5db8f1152853 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:23.374237227 +0000 UTC m=+2.029354371 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs") pod "network-metrics-daemon-tt8gq" (UID: "a1f5a77d-43d5-48bc-a8b5-5db8f1152853") : secret "metrics-daemon-secret" not found Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874272 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6cd67a78-09f1-4988-99b0-1ac9097f2f39-device-dir\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874284 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-host-run-netns\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874326 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72f98900-ff17-447f-8429-60c3e47fc7d1-metrics-tls\") pod \"dns-default-hxw5f\" (UID: \"72f98900-ff17-447f-8429-60c3e47fc7d1\") " pod="openshift-dns/dns-default-hxw5f" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874340 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-etc-modprobe-d\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874346 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67ae987f-79b9-4852-98bd-e0537c412b17-host-run-netns\") pod \"multus-qh47z\" (UID: \"67ae987f-79b9-4852-98bd-e0537c412b17\") " pod="openshift-multus/multus-qh47z" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874354 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-etc-sysctl-d\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874382 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-log-socket\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874412 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6cd67a78-09f1-4988-99b0-1ac9097f2f39-registration-dir\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874451 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-log-socket\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874469 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-etc-sysctl-d\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874472 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6cd67a78-09f1-4988-99b0-1ac9097f2f39-registration-dir\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874498 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/375debb4-cfe1-44ab-907b-0a34a031c2ea-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7grdp\" (UID: \"375debb4-cfe1-44ab-907b-0a34a031c2ea\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7grdp" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874527 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-cni-bin\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874553 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-etc-sysctl-conf\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874589 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-host-cni-bin\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:22.874617 5040 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:22.874648 5040 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/375debb4-cfe1-44ab-907b-0a34a031c2ea-networking-console-plugin-cert podName:375debb4-cfe1-44ab-907b-0a34a031c2ea nodeName:}" failed. No retries permitted until 2026-11-01 05:45:23.374638771 +0000 UTC m=+2.029755920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/375debb4-cfe1-44ab-907b-0a34a031c2ea-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7grdp" (UID: "375debb4-cfe1-44ab-907b-0a34a031c2ea") : secret "networking-console-plugin-cert" not found Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874677 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-registry-tls\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874679 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-etc-sysctl-conf\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874696 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5705a40-d3e4-40a8-a561-17c435f388b1-cnibin\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874714 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-etc-kubernetes\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874730 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7dc7486d-3188-4bb8-b907-b653caa18660-kubelet-config\") pod \"global-pull-secret-syncer-r5l82\" (UID: \"7dc7486d-3188-4bb8-b907-b653caa18660\") " pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874780 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5705a40-d3e4-40a8-a561-17c435f388b1-cnibin\") pod \"multus-additional-cni-plugins-b7rrq\" (UID: \"f5705a40-d3e4-40a8-a561-17c435f388b1\") " pod="openshift-multus/multus-additional-cni-plugins-b7rrq" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874782 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-node-log\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874812 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/21519345-86b8-441e-a395-1e8385cd3732-node-log\") pod \"ovnkube-node-7bgdz\" (UID: \"21519345-86b8-441e-a395-1e8385cd3732\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874801 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2acbc647-2361-4935-b503-5bd9af74f822-etc-kubernetes\") pod \"tuned-cb599\" (UID: \"2acbc647-2361-4935-b503-5bd9af74f822\") " pod="openshift-cluster-node-tuning-operator/tuned-cb599" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874820 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6cd67a78-09f1-4988-99b0-1ac9097f2f39-etc-selinux\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874838 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7dc7486d-3188-4bb8-b907-b653caa18660-kubelet-config\") pod \"global-pull-secret-syncer-r5l82\" (UID: \"7dc7486d-3188-4bb8-b907-b653caa18660\") " pod="kube-system/global-pull-secret-syncer-r5l82" Nov 01 05:45:22.875187 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.874874 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6cd67a78-09f1-4988-99b0-1ac9097f2f39-etc-selinux\") pod \"aws-ebs-csi-driver-node-clt4x\" (UID: \"6cd67a78-09f1-4988-99b0-1ac9097f2f39\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clt4x" Nov 01 05:45:22.877701 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.875867 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-registry-tls\") pod \"image-registry-5f55979cbb-ngg8z\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:22.877701 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.876423 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72f98900-ff17-447f-8429-60c3e47fc7d1-metrics-tls\") pod \"dns-default-hxw5f\" (UID: \"72f98900-ff17-447f-8429-60c3e47fc7d1\") " pod="openshift-dns/dns-default-hxw5f" Nov 01 05:45:22.877701 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.877291 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-registry-tls\") pod \"image-registry-847d56945f-lt2tl\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:22.909668 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.909641 5040 scope.go:122] "RemoveContainer" containerID="aeacbab6a94ca1ae2b5b61836714026e446cbb0d816488b4c6374f81be6410c8" Nov 01 05:45:22.910614 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.910596 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:22.918271 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.918251 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:22.919470 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.919449 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hxw5f" Nov 01 05:45:22.922273 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:22.922227 5040 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 05:45:23.196807 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:23.196781 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2qtwx_ec7c5307-5d9b-4b92-b4e3-35d964ba4cd8/dns-node-resolver/0.log" Nov 01 05:45:23.234236 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:23.234170 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:23.385150 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:23.385124 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-metrics-certs\") pod \"router-default-6c5b5d5bd5-776bc\" (UID: \"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78\") " pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:23.385336 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:23.385160 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs\") pod \"network-metrics-daemon-tt8gq\" (UID: \"a1f5a77d-43d5-48bc-a8b5-5db8f1152853\") " pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:45:23.385336 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:23.385194 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/375debb4-cfe1-44ab-907b-0a34a031c2ea-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7grdp\" (UID: \"375debb4-cfe1-44ab-907b-0a34a031c2ea\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7grdp" Nov 01 05:45:23.385336 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:23.385233 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2191a9b0-b911-46e9-8ed4-b5119e37341d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54tr8\" (UID: \"2191a9b0-b911-46e9-8ed4-b5119e37341d\") " pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:23.385336 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:23.385251 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc9422c2-b041-4f6b-8495-961a9571c1e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9j8j4\" (UID: \"dc9422c2-b041-4f6b-8495-961a9571c1e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9j8j4" Nov 01 05:45:23.385336 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:23.385288 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-flt78\" (UID: \"320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flt78" Nov 01 05:45:23.385336 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:23.385305 5040 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 01 05:45:23.385595 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:23.385341 5040 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 05:45:23.385595 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:23.385350 5040 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 05:45:23.385595 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:23.385379 5040 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs podName:a1f5a77d-43d5-48bc-a8b5-5db8f1152853 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:24.385360455 +0000 UTC m=+3.040477624 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs") pod "network-metrics-daemon-tt8gq" (UID: "a1f5a77d-43d5-48bc-a8b5-5db8f1152853") : secret "metrics-daemon-secret" not found Nov 01 05:45:23.385595 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:23.385386 5040 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 05:45:23.385595 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:23.385395 5040 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 05:45:23.385595 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:23.385397 5040 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/375debb4-cfe1-44ab-907b-0a34a031c2ea-networking-console-plugin-cert podName:375debb4-cfe1-44ab-907b-0a34a031c2ea nodeName:}" failed. No retries permitted until 2026-11-01 05:45:24.385388446 +0000 UTC m=+3.040505595 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/375debb4-cfe1-44ab-907b-0a34a031c2ea-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7grdp" (UID: "375debb4-cfe1-44ab-907b-0a34a031c2ea") : secret "networking-console-plugin-cert" not found Nov 01 05:45:23.385595 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:23.385427 5040 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 05:45:23.385595 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:23.385443 5040 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2191a9b0-b911-46e9-8ed4-b5119e37341d-insights-runtime-extractor-tls podName:2191a9b0-b911-46e9-8ed4-b5119e37341d nodeName:}" failed. No retries permitted until 2026-11-01 05:45:24.385433128 +0000 UTC m=+3.040550273 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2191a9b0-b911-46e9-8ed4-b5119e37341d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-54tr8" (UID: "2191a9b0-b911-46e9-8ed4-b5119e37341d") : secret "insights-runtime-extractor-tls" not found Nov 01 05:45:23.385595 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:23.385455 5040 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc9422c2-b041-4f6b-8495-961a9571c1e4-cluster-monitoring-operator-tls podName:dc9422c2-b041-4f6b-8495-961a9571c1e4 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:24.385448653 +0000 UTC m=+3.040565809 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dc9422c2-b041-4f6b-8495-961a9571c1e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9j8j4" (UID: "dc9422c2-b041-4f6b-8495-961a9571c1e4") : secret "cluster-monitoring-operator-tls" not found Nov 01 05:45:23.385595 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:23.385464 5040 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-samples-operator-tls podName:320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:24.385459584 +0000 UTC m=+3.040576729 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-flt78" (UID: "320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2") : secret "samples-operator-tls" not found Nov 01 05:45:23.385595 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:23.385474 5040 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-metrics-certs podName:fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:24.385468831 +0000 UTC m=+3.040585975 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-metrics-certs") pod "router-default-6c5b5d5bd5-776bc" (UID: "fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78") : secret "router-metrics-certs-default" not found Nov 01 05:45:23.388010 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:23.387987 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hxw5f"] Nov 01 05:45:23.390965 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:23.390929 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f98900_ff17_447f_8429_60c3e47fc7d1.slice/crio-2b35cec339a44840478adb1344ebc03e4de8f81e2ea578058dc5f4fd13c09bc7 WatchSource:0}: Error finding container 2b35cec339a44840478adb1344ebc03e4de8f81e2ea578058dc5f4fd13c09bc7: Status 404 returned error can't find the container with id 2b35cec339a44840478adb1344ebc03e4de8f81e2ea578058dc5f4fd13c09bc7 Nov 01 05:45:23.391342 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:23.391328 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f55979cbb-ngg8z"] Nov 01 05:45:23.428863 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:23.428833 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9ccbc38_43f9_4396_bb59_2f746b3d639d.slice/crio-006608d8bd74a5415ebf37346ec8b061ea51019dcbdcc3df8eb2525c696efcc5 WatchSource:0}: Error finding container 006608d8bd74a5415ebf37346ec8b061ea51019dcbdcc3df8eb2525c696efcc5: Status 404 returned error can't find the container with id 006608d8bd74a5415ebf37346ec8b061ea51019dcbdcc3df8eb2525c696efcc5 Nov 01 05:45:23.441900 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:23.441829 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-847d56945f-lt2tl"] Nov 01 05:45:23.445657 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:23.445633 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7729d415_70e5_4be3_a9cc_bbd13e065c71.slice/crio-6d613cf316fe8124c019364ee2e2e8ce200e854e8ba730eb54c48c6553d622b2 WatchSource:0}: Error finding container 6d613cf316fe8124c019364ee2e2e8ce200e854e8ba730eb54c48c6553d622b2: Status 404 returned error can't find the container with id 6d613cf316fe8124c019364ee2e2e8ce200e854e8ba730eb54c48c6553d622b2 Nov 01 05:45:23.532708 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:23.532673 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:23.593344 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:23.593301 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" event={"ID":"f9ccbc38-43f9-4396-bb59-2f746b3d639d","Type":"ContainerStarted","Data":"b9edaa694b100ff6888b4eddd67c6018f975cab63b85b6a0a59e21431ea5e61d"} Nov 01 05:45:23.593543 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:23.593368 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" event={"ID":"f9ccbc38-43f9-4396-bb59-2f746b3d639d","Type":"ContainerStarted","Data":"006608d8bd74a5415ebf37346ec8b061ea51019dcbdcc3df8eb2525c696efcc5"} Nov 01 05:45:23.594950 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:23.594928 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 05:45:23.595406 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:23.595387 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/1.log" Nov 01 05:45:23.595473 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:23.595429 5040 generic.go:356] "Generic (PLEG): container finished" podID="580bebeb-9495-4541-8d1d-3bf6fef1a061" containerID="6e42d58210f0da4839afe6b57b05e3f7cd31735668a53a13e108e8906795e026" exitCode=255 Nov 01 05:45:23.595560 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:23.595542 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" event={"ID":"580bebeb-9495-4541-8d1d-3bf6fef1a061","Type":"ContainerDied","Data":"6e42d58210f0da4839afe6b57b05e3f7cd31735668a53a13e108e8906795e026"} Nov 01 05:45:23.595608 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:23.595593 5040 scope.go:122] "RemoveContainer" containerID="aeacbab6a94ca1ae2b5b61836714026e446cbb0d816488b4c6374f81be6410c8" Nov 01 05:45:23.595869 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:23.595854 5040 scope.go:122] "RemoveContainer" containerID="6e42d58210f0da4839afe6b57b05e3f7cd31735668a53a13e108e8906795e026" Nov 01 05:45:23.596066 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:23.596043 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-njjn4_openshift-console-operator(580bebeb-9495-4541-8d1d-3bf6fef1a061)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" podUID="580bebeb-9495-4541-8d1d-3bf6fef1a061" Nov 01 05:45:23.597336 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:23.597313 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-847d56945f-lt2tl" event={"ID":"7729d415-70e5-4be3-a9cc-bbd13e065c71","Type":"ContainerStarted","Data":"246378f47402702a056c8ddc696564bf27ccfe280249c02d5e7b95008b58f5c6"} Nov 01 05:45:23.597427 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:23.597354 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-847d56945f-lt2tl" event={"ID":"7729d415-70e5-4be3-a9cc-bbd13e065c71","Type":"ContainerStarted","Data":"6d613cf316fe8124c019364ee2e2e8ce200e854e8ba730eb54c48c6553d622b2"} Nov 01 05:45:23.598834 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:23.598812 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hxw5f" event={"ID":"72f98900-ff17-447f-8429-60c3e47fc7d1","Type":"ContainerStarted","Data":"2b35cec339a44840478adb1344ebc03e4de8f81e2ea578058dc5f4fd13c09bc7"} Nov 01 05:45:23.797660 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:23.797590 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5f55979cbb-ngg8z_f9ccbc38-43f9-4396-bb59-2f746b3d639d/registry/0.log" Nov 01 05:45:23.997234 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:23.997208 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-847d56945f-lt2tl_7729d415-70e5-4be3-a9cc-bbd13e065c71/registry/0.log" Nov 01 05:45:24.395904 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:24.395869 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs\") pod \"network-metrics-daemon-tt8gq\" (UID: \"a1f5a77d-43d5-48bc-a8b5-5db8f1152853\") " pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:45:24.396071 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:24.395941 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/375debb4-cfe1-44ab-907b-0a34a031c2ea-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7grdp\" (UID: \"375debb4-cfe1-44ab-907b-0a34a031c2ea\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7grdp" Nov 01 05:45:24.396071 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:24.395962 5040 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 01 05:45:24.396071 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:24.395997 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2191a9b0-b911-46e9-8ed4-b5119e37341d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54tr8\" (UID: \"2191a9b0-b911-46e9-8ed4-b5119e37341d\") " pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:24.396071 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:24.396017 5040 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs podName:a1f5a77d-43d5-48bc-a8b5-5db8f1152853 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:26.395998998 +0000 UTC m=+5.051116173 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs") pod "network-metrics-daemon-tt8gq" (UID: "a1f5a77d-43d5-48bc-a8b5-5db8f1152853") : secret "metrics-daemon-secret" not found Nov 01 05:45:24.396071 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:24.396052 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc9422c2-b041-4f6b-8495-961a9571c1e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9j8j4\" (UID: \"dc9422c2-b041-4f6b-8495-961a9571c1e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9j8j4" Nov 01 05:45:24.396342 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:24.396069 5040 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 05:45:24.396342 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:24.396087 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-flt78\" (UID: \"320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flt78" Nov 01 05:45:24.396342 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:24.396127 5040 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/375debb4-cfe1-44ab-907b-0a34a031c2ea-networking-console-plugin-cert podName:375debb4-cfe1-44ab-907b-0a34a031c2ea nodeName:}" failed. No retries permitted until 2026-11-01 05:45:26.396111056 +0000 UTC m=+5.051228215 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/375debb4-cfe1-44ab-907b-0a34a031c2ea-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7grdp" (UID: "375debb4-cfe1-44ab-907b-0a34a031c2ea") : secret "networking-console-plugin-cert" not found Nov 01 05:45:24.396342 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:24.396071 5040 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 05:45:24.396342 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:24.396147 5040 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 05:45:24.396342 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:24.396160 5040 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 05:45:24.396342 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:24.396185 5040 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2191a9b0-b911-46e9-8ed4-b5119e37341d-insights-runtime-extractor-tls podName:2191a9b0-b911-46e9-8ed4-b5119e37341d nodeName:}" failed. No retries permitted until 2026-11-01 05:45:26.396172697 +0000 UTC m=+5.051289854 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2191a9b0-b911-46e9-8ed4-b5119e37341d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-54tr8" (UID: "2191a9b0-b911-46e9-8ed4-b5119e37341d") : secret "insights-runtime-extractor-tls" not found Nov 01 05:45:24.396342 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:24.396182 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-metrics-certs\") pod \"router-default-6c5b5d5bd5-776bc\" (UID: \"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78\") " pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:24.396342 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:24.396242 5040 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc9422c2-b041-4f6b-8495-961a9571c1e4-cluster-monitoring-operator-tls podName:dc9422c2-b041-4f6b-8495-961a9571c1e4 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:26.396203588 +0000 UTC m=+5.051320739 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dc9422c2-b041-4f6b-8495-961a9571c1e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9j8j4" (UID: "dc9422c2-b041-4f6b-8495-961a9571c1e4") : secret "cluster-monitoring-operator-tls" not found Nov 01 05:45:24.396342 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:24.396242 5040 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 05:45:24.396342 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:24.396300 5040 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-samples-operator-tls podName:320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:26.39629114 +0000 UTC m=+5.051408285 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-flt78" (UID: "320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2") : secret "samples-operator-tls" not found Nov 01 05:45:24.396342 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:24.396314 5040 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-metrics-certs podName:fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:26.396307921 +0000 UTC m=+5.051425066 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-metrics-certs") pod "router-default-6c5b5d5bd5-776bc" (UID: "fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78") : secret "router-metrics-certs-default" not found Nov 01 05:45:24.397040 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:24.397020 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tw96g_cde21439-a494-47cd-8cc5-161fc96a6701/node-ca/0.log" Nov 01 05:45:24.603091 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:24.603064 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 05:45:24.604661 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:24.604628 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n6nns" event={"ID":"baad0c0a-7a5d-454d-8ea9-4a3b27115992","Type":"ContainerStarted","Data":"d0156d3148b4eeae21f061eb5030f0ab9f789dbd4fa09863ebb768dce302ea3c"} Nov 01 05:45:24.606132 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:24.606107 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wtlk8" event={"ID":"7db79928-e03b-4027-9cca-40191f6a300d","Type":"ContainerStarted","Data":"c40aef1c58067d93c9aaddfe4a429c26f102daf7e8b0dba19d192b0cd3a89e18"} Nov 01 05:45:24.606132 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:24.606132 5040 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 05:45:24.606317 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:24.606150 5040 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 05:45:24.606317 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:24.606150 5040 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 05:45:24.996926 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:24.996894 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dwklx_1b19c3b0-887e-4e27-bac3-028003f1aac5/serve-healthcheck-canary/0.log" Nov 01 05:45:25.173359 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:25.173323 5040 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" Nov 01 05:45:25.173744 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:25.173730 5040 scope.go:122] "RemoveContainer" containerID="6e42d58210f0da4839afe6b57b05e3f7cd31735668a53a13e108e8906795e026" Nov 01 05:45:25.173945 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:25.173926 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-njjn4_openshift-console-operator(580bebeb-9495-4541-8d1d-3bf6fef1a061)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" podUID="580bebeb-9495-4541-8d1d-3bf6fef1a061" Nov 01 05:45:25.197610 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:25.197564 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-n6nns_baad0c0a-7a5d-454d-8ea9-4a3b27115992/migrator/0.log" Nov 01 05:45:25.397275 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:25.397201 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-wtlk8_7db79928-e03b-4027-9cca-40191f6a300d/migrator/0.log" Nov 01 05:45:25.537244 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:25.537206 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ljcjx" Nov 01 05:45:25.598049 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:25.598021 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-95sds_109327c0-f8f8-4dde-9cfe-489bf5a7eaad/kube-storage-version-migrator-operator/0.log" Nov 01 05:45:26.413789 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:26.413745 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/375debb4-cfe1-44ab-907b-0a34a031c2ea-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7grdp\" (UID: \"375debb4-cfe1-44ab-907b-0a34a031c2ea\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7grdp" Nov 01 05:45:26.414212 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:26.413804 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2191a9b0-b911-46e9-8ed4-b5119e37341d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54tr8\" (UID: \"2191a9b0-b911-46e9-8ed4-b5119e37341d\") " pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:26.414212 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:26.413826 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc9422c2-b041-4f6b-8495-961a9571c1e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9j8j4\" (UID: \"dc9422c2-b041-4f6b-8495-961a9571c1e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9j8j4" Nov 01 05:45:26.414212 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:26.413848 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-flt78\" (UID: \"320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flt78" Nov 01 05:45:26.414212 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:26.413879 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-metrics-certs\") pod \"router-default-6c5b5d5bd5-776bc\" (UID: \"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78\") " pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:26.414212 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:26.413898 5040 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 05:45:26.414212 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:26.413960 5040 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 05:45:26.414212 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:26.413964 5040 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 05:45:26.414212 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:26.413977 5040 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/375debb4-cfe1-44ab-907b-0a34a031c2ea-networking-console-plugin-cert podName:375debb4-cfe1-44ab-907b-0a34a031c2ea nodeName:}" failed. No retries permitted until 2026-11-01 05:45:30.413957962 +0000 UTC m=+9.069075120 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/375debb4-cfe1-44ab-907b-0a34a031c2ea-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7grdp" (UID: "375debb4-cfe1-44ab-907b-0a34a031c2ea") : secret "networking-console-plugin-cert" not found Nov 01 05:45:26.414212 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:26.413994 5040 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 05:45:26.414212 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:26.413997 5040 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-samples-operator-tls podName:320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:30.413987235 +0000 UTC m=+9.069104395 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-flt78" (UID: "320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2") : secret "samples-operator-tls" not found Nov 01 05:45:26.414212 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:26.414016 5040 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2191a9b0-b911-46e9-8ed4-b5119e37341d-insights-runtime-extractor-tls podName:2191a9b0-b911-46e9-8ed4-b5119e37341d nodeName:}" failed. No retries permitted until 2026-11-01 05:45:30.41400428 +0000 UTC m=+9.069121425 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2191a9b0-b911-46e9-8ed4-b5119e37341d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-54tr8" (UID: "2191a9b0-b911-46e9-8ed4-b5119e37341d") : secret "insights-runtime-extractor-tls" not found Nov 01 05:45:26.414212 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:26.413907 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs\") pod \"network-metrics-daemon-tt8gq\" (UID: \"a1f5a77d-43d5-48bc-a8b5-5db8f1152853\") " pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:45:26.414212 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:26.414030 5040 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-metrics-certs podName:fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:30.414023978 +0000 UTC m=+9.069141123 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-metrics-certs") pod "router-default-6c5b5d5bd5-776bc" (UID: "fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78") : secret "router-metrics-certs-default" not found Nov 01 05:45:26.414212 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:26.413962 5040 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 01 05:45:26.414212 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:26.414080 5040 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs podName:a1f5a77d-43d5-48bc-a8b5-5db8f1152853 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:30.414069618 +0000 UTC m=+9.069186769 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs") pod "network-metrics-daemon-tt8gq" (UID: "a1f5a77d-43d5-48bc-a8b5-5db8f1152853") : secret "metrics-daemon-secret" not found Nov 01 05:45:26.414212 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:26.414039 5040 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 05:45:26.414212 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:26.414118 5040 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc9422c2-b041-4f6b-8495-961a9571c1e4-cluster-monitoring-operator-tls podName:dc9422c2-b041-4f6b-8495-961a9571c1e4 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:30.414109955 +0000 UTC m=+9.069227146 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dc9422c2-b041-4f6b-8495-961a9571c1e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9j8j4" (UID: "dc9422c2-b041-4f6b-8495-961a9571c1e4") : secret "cluster-monitoring-operator-tls" not found Nov 01 05:45:26.616014 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:26.615977 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hxw5f" event={"ID":"72f98900-ff17-447f-8429-60c3e47fc7d1","Type":"ContainerStarted","Data":"1dfdd37076c54e970f8c3987dbae1b65a13f1e4b9aa04c9dde03d76ac7b6c295"} Nov 01 05:45:26.616218 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:26.616028 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hxw5f" event={"ID":"72f98900-ff17-447f-8429-60c3e47fc7d1","Type":"ContainerStarted","Data":"75aac01ba5e01179b3fd7715c86f3a8aaa874b3797bfc862e7d7dac5ce327c3f"} Nov 01 05:45:26.616218 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:26.616087 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hxw5f" Nov 01 05:45:29.201018 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:29.200979 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:30.341391 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:30.341354 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:30.448741 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:30.448704 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2191a9b0-b911-46e9-8ed4-b5119e37341d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54tr8\" (UID: \"2191a9b0-b911-46e9-8ed4-b5119e37341d\") " pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:30.448741 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:30.448742 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc9422c2-b041-4f6b-8495-961a9571c1e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9j8j4\" (UID: \"dc9422c2-b041-4f6b-8495-961a9571c1e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9j8j4" Nov 01 05:45:30.448986 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:30.448770 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-flt78\" (UID: \"320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flt78" Nov 01 05:45:30.448986 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:30.448807 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-metrics-certs\") pod \"router-default-6c5b5d5bd5-776bc\" (UID: \"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78\") " pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:30.448986 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:30.448847 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs\") pod \"network-metrics-daemon-tt8gq\" (UID: \"a1f5a77d-43d5-48bc-a8b5-5db8f1152853\") " pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:45:30.448986 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:30.448857 5040 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 05:45:30.448986 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:30.448943 5040 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2191a9b0-b911-46e9-8ed4-b5119e37341d-insights-runtime-extractor-tls podName:2191a9b0-b911-46e9-8ed4-b5119e37341d nodeName:}" failed. No retries permitted until 2026-11-01 05:45:38.448922091 +0000 UTC m=+17.104039250 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2191a9b0-b911-46e9-8ed4-b5119e37341d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-54tr8" (UID: "2191a9b0-b911-46e9-8ed4-b5119e37341d") : secret "insights-runtime-extractor-tls" not found Nov 01 05:45:30.448986 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:30.448969 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/375debb4-cfe1-44ab-907b-0a34a031c2ea-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7grdp\" (UID: \"375debb4-cfe1-44ab-907b-0a34a031c2ea\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7grdp" Nov 01 05:45:30.449462 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:30.449441 5040 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 01 05:45:30.449568 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:30.449536 5040 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs podName:a1f5a77d-43d5-48bc-a8b5-5db8f1152853 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:38.44951852 +0000 UTC m=+17.104635683 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs") pod "network-metrics-daemon-tt8gq" (UID: "a1f5a77d-43d5-48bc-a8b5-5db8f1152853") : secret "metrics-daemon-secret" not found Nov 01 05:45:30.451292 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:30.451262 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc9422c2-b041-4f6b-8495-961a9571c1e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9j8j4\" (UID: \"dc9422c2-b041-4f6b-8495-961a9571c1e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9j8j4" Nov 01 05:45:30.451596 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:30.451576 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-flt78\" (UID: \"320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flt78" Nov 01 05:45:30.451731 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:30.451713 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/375debb4-cfe1-44ab-907b-0a34a031c2ea-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7grdp\" (UID: \"375debb4-cfe1-44ab-907b-0a34a031c2ea\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7grdp" Nov 01 05:45:30.451799 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:30.451780 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78-metrics-certs\") pod \"router-default-6c5b5d5bd5-776bc\" (UID: \"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78\") " pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:30.711785 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:30.711697 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:30.711949 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:30.711797 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flt78" Nov 01 05:45:30.711949 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:30.711843 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9j8j4" Nov 01 05:45:30.712722 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:30.712708 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-7grdp" Nov 01 05:45:30.962345 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:30.962256 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-n7mks" Nov 01 05:45:31.195495 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:31.195450 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-7grdp"] Nov 01 05:45:31.198172 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:31.198143 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod375debb4_cfe1_44ab_907b_0a34a031c2ea.slice/crio-6379816a867e1b8909b381d2c02c92c81deb50af6c8a82ce2b1fbd2b35e02484 WatchSource:0}: Error finding container 6379816a867e1b8909b381d2c02c92c81deb50af6c8a82ce2b1fbd2b35e02484: Status 404 returned error can't find the container with id 6379816a867e1b8909b381d2c02c92c81deb50af6c8a82ce2b1fbd2b35e02484 Nov 01 05:45:31.198680 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:31.198579 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-9j8j4"] Nov 01 05:45:31.237405 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:31.237372 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc9422c2_b041_4f6b_8495_961a9571c1e4.slice/crio-5bc54ee3057b4c8a63af3f5441249b09cd3246371ffce6108987f167ddb4173f WatchSource:0}: Error finding container 5bc54ee3057b4c8a63af3f5441249b09cd3246371ffce6108987f167ddb4173f: Status 404 returned error can't find the container with id 5bc54ee3057b4c8a63af3f5441249b09cd3246371ffce6108987f167ddb4173f Nov 01 05:45:31.246200 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:31.246172 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flt78"] Nov 01 05:45:31.251108 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:31.251079 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6c5b5d5bd5-776bc"] Nov 01 05:45:31.271246 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:31.271223 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" Nov 01 05:45:31.271613 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:31.271599 5040 scope.go:122] "RemoveContainer" containerID="6e42d58210f0da4839afe6b57b05e3f7cd31735668a53a13e108e8906795e026" Nov 01 05:45:31.271797 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:45:31.271778 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-njjn4_openshift-console-operator(580bebeb-9495-4541-8d1d-3bf6fef1a061)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" podUID="580bebeb-9495-4541-8d1d-3bf6fef1a061" Nov 01 05:45:31.473247 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:31.473222 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7bgdz" Nov 01 05:45:31.634559 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:31.634528 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9j8j4" event={"ID":"dc9422c2-b041-4f6b-8495-961a9571c1e4","Type":"ContainerStarted","Data":"5bc54ee3057b4c8a63af3f5441249b09cd3246371ffce6108987f167ddb4173f"} Nov 01 05:45:31.635734 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:31.635707 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-7grdp" event={"ID":"375debb4-cfe1-44ab-907b-0a34a031c2ea","Type":"ContainerStarted","Data":"6379816a867e1b8909b381d2c02c92c81deb50af6c8a82ce2b1fbd2b35e02484"} Nov 01 05:45:31.636991 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:31.636955 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flt78" event={"ID":"320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2","Type":"ContainerStarted","Data":"e43d7823d4067c3ab32db8623f4081ac66361e27e001a27152866cf1d3acea85"} Nov 01 05:45:31.638451 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:31.638427 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" event={"ID":"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78","Type":"ContainerStarted","Data":"054dd3143767bfb1e65a8ef09f7e6f6b4ed28d2ab4458221f9a3dac1a8175e26"} Nov 01 05:45:31.638566 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:31.638460 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" event={"ID":"fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78","Type":"ContainerStarted","Data":"29d9c49a81ca373d37e8204907855a2d0ceed62e5c264bfa8a0c8a212b3bc94e"} Nov 01 05:45:31.711950 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:31.711912 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:31.714715 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:31.714684 5040 patch_prober.go:29] interesting pod/router-default-6c5b5d5bd5-776bc container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Nov 01 05:45:31.714715 ip-10-0-143-195 kubenswrapper[5040]: [-]has-synced failed: reason withheld Nov 01 05:45:31.714715 ip-10-0-143-195 kubenswrapper[5040]: [+]process-running ok Nov 01 05:45:31.714715 ip-10-0-143-195 kubenswrapper[5040]: healthz check failed Nov 01 05:45:31.714923 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:31.714741 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" podUID="fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Nov 01 05:45:32.713970 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:32.713708 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:33.507955 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:33.507925 5040 scope.go:122] "RemoveContainer" containerID="6e42d58210f0da4839afe6b57b05e3f7cd31735668a53a13e108e8906795e026" Nov 01 05:45:33.650308 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:33.650287 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 05:45:33.650430 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:33.650406 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" event={"ID":"580bebeb-9495-4541-8d1d-3bf6fef1a061","Type":"ContainerStarted","Data":"1cb1a0c9533c2e7a9e20d04cb9d5947d17738c4fac3278631d978b4b56e9dd1c"} Nov 01 05:45:33.651523 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:33.651256 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" Nov 01 05:45:33.651697 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:33.651671 5040 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-njjn4 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.12:8443/readyz\": dial tcp 10.132.0.12:8443: connect: connection refused" start-of-body= Nov 01 05:45:33.651782 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:33.651718 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" podUID="580bebeb-9495-4541-8d1d-3bf6fef1a061" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.12:8443/readyz\": dial tcp 10.132.0.12:8443: connect: connection refused" Nov 01 05:45:33.653808 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:33.653786 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-7grdp" event={"ID":"375debb4-cfe1-44ab-907b-0a34a031c2ea","Type":"ContainerStarted","Data":"74387d20e20a3850220e80b21bc27b4674e61361b048aa3e2251516eec9a376d"} Nov 01 05:45:33.653913 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:33.653829 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:33.655099 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:33.655073 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6c5b5d5bd5-776bc" Nov 01 05:45:34.662193 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:34.662141 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9j8j4" event={"ID":"dc9422c2-b041-4f6b-8495-961a9571c1e4","Type":"ContainerStarted","Data":"74549f220d6b371481c41fc2c3c14940d1f0652c13e13734fadb91e998ca9198"} Nov 01 05:45:34.664871 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:34.664836 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flt78" event={"ID":"320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2","Type":"ContainerStarted","Data":"6fecd2d0a60dd7a40a42b52aecceba4143ae90d966f11e756553fb9d1d04fd0f"} Nov 01 05:45:34.665011 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:34.664892 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flt78" event={"ID":"320c5f5d-fa1c-4e5d-9fc6-b2c3c0dcbdc2","Type":"ContainerStarted","Data":"94e50bd9d3ca5233553e21ad45df9f42358facee4613044b12c1f13792dd6820"} Nov 01 05:45:34.669108 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:34.669082 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-njjn4" Nov 01 05:45:36.621670 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:36.621639 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hxw5f" Nov 01 05:45:37.583035 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:37.582998 5040 patch_prober.go:29] interesting pod/image-registry-5f55979cbb-ngg8z container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 01 05:45:37.583178 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:37.583049 5040 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" podUID="f9ccbc38-43f9-4396-bb59-2f746b3d639d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 05:45:38.524852 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:38.524812 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2191a9b0-b911-46e9-8ed4-b5119e37341d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54tr8\" (UID: \"2191a9b0-b911-46e9-8ed4-b5119e37341d\") " pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:38.525247 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:38.524884 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs\") pod \"network-metrics-daemon-tt8gq\" (UID: \"a1f5a77d-43d5-48bc-a8b5-5db8f1152853\") " pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:45:38.527340 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:38.527310 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2191a9b0-b911-46e9-8ed4-b5119e37341d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54tr8\" (UID: \"2191a9b0-b911-46e9-8ed4-b5119e37341d\") " pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:38.527444 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:38.527351 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1f5a77d-43d5-48bc-a8b5-5db8f1152853-metrics-certs\") pod \"network-metrics-daemon-tt8gq\" (UID: \"a1f5a77d-43d5-48bc-a8b5-5db8f1152853\") " pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:45:38.813472 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:38.813373 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt8gq" Nov 01 05:45:38.820285 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:38.820259 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-54tr8" Nov 01 05:45:39.041520 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:39.041468 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f55979cbb-ngg8z"] Nov 01 05:45:39.045977 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:39.045943 5040 patch_prober.go:29] interesting pod/image-registry-5f55979cbb-ngg8z container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 01 05:45:39.046134 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:39.046007 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" podUID="f9ccbc38-43f9-4396-bb59-2f746b3d639d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 05:45:39.184552 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:39.184529 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tt8gq"] Nov 01 05:45:39.187262 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:39.187230 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1f5a77d_43d5_48bc_a8b5_5db8f1152853.slice/crio-3f5d5b91445c663da91b390431f6f41fbc8b74493d1b935a2ea45ab2093beeaa WatchSource:0}: Error finding container 3f5d5b91445c663da91b390431f6f41fbc8b74493d1b935a2ea45ab2093beeaa: Status 404 returned error can't find the container with id 3f5d5b91445c663da91b390431f6f41fbc8b74493d1b935a2ea45ab2093beeaa Nov 01 05:45:39.239182 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:39.239027 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-54tr8"] Nov 01 05:45:39.241949 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:39.241918 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2191a9b0_b911_46e9_8ed4_b5119e37341d.slice/crio-352f66967b12831551cf6de9d9a625d6213d9848206efab66b7720860c3cf223 WatchSource:0}: Error finding container 352f66967b12831551cf6de9d9a625d6213d9848206efab66b7720860c3cf223: Status 404 returned error can't find the container with id 352f66967b12831551cf6de9d9a625d6213d9848206efab66b7720860c3cf223 Nov 01 05:45:39.684944 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:39.684904 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-54tr8" event={"ID":"2191a9b0-b911-46e9-8ed4-b5119e37341d","Type":"ContainerStarted","Data":"fa4dba0e23b647b7845e35982e65dcfa3189cf788c6ff05e6a91dd19c290a95e"} Nov 01 05:45:39.685373 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:39.684969 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-54tr8" event={"ID":"2191a9b0-b911-46e9-8ed4-b5119e37341d","Type":"ContainerStarted","Data":"352f66967b12831551cf6de9d9a625d6213d9848206efab66b7720860c3cf223"} Nov 01 05:45:39.686076 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:39.686048 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tt8gq" event={"ID":"a1f5a77d-43d5-48bc-a8b5-5db8f1152853","Type":"ContainerStarted","Data":"3f5d5b91445c663da91b390431f6f41fbc8b74493d1b935a2ea45ab2093beeaa"} Nov 01 05:45:40.348727 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:40.348381 5040 patch_prober.go:29] interesting pod/image-registry-847d56945f-lt2tl container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 01 05:45:40.348727 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:40.348444 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-847d56945f-lt2tl" podUID="7729d415-70e5-4be3-a9cc-bbd13e065c71" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 05:45:40.694077 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:40.693985 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-54tr8" event={"ID":"2191a9b0-b911-46e9-8ed4-b5119e37341d","Type":"ContainerStarted","Data":"e02d83405ec5854c7b7d22e992e182ca71dcb18abee9e8c7e9b1520c664beeed"} Nov 01 05:45:41.701008 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:41.700973 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tt8gq" event={"ID":"a1f5a77d-43d5-48bc-a8b5-5db8f1152853","Type":"ContainerStarted","Data":"ff61ddc433789da3b9c070edfbde97508e42d0116462043040968f8c8193a624"} Nov 01 05:45:41.701499 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:41.701024 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tt8gq" event={"ID":"a1f5a77d-43d5-48bc-a8b5-5db8f1152853","Type":"ContainerStarted","Data":"5815b1fdee5cad14f16fbde91aa8a11911c8e3e9ded67d90460c16b693409fa1"} Nov 01 05:45:42.068300 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:42.068247 5040 patch_prober.go:29] interesting pod/image-registry-847d56945f-lt2tl container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 01 05:45:42.068498 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:42.068312 5040 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-847d56945f-lt2tl" podUID="7729d415-70e5-4be3-a9cc-bbd13e065c71" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 05:45:42.707936 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:42.707851 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-54tr8" event={"ID":"2191a9b0-b911-46e9-8ed4-b5119e37341d","Type":"ContainerStarted","Data":"3e4bd1e3cd3ce5f3497be988d02cf6d7a589ebcd6a610962e8946da4bd213379"} Nov 01 05:45:49.046421 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:49.046389 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:45:49.291823 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:49.291792 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2k242"] Nov 01 05:45:49.318521 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:49.318423 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2k242" Nov 01 05:45:49.423372 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:49.423334 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-899zk\" (UniqueName: \"kubernetes.io/projected/4137336d-3d4e-40e1-8d59-f32700970597-kube-api-access-899zk\") pod \"node-exporter-2k242\" (UID: \"4137336d-3d4e-40e1-8d59-f32700970597\") " pod="openshift-monitoring/node-exporter-2k242" Nov 01 05:45:49.423372 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:49.423375 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4137336d-3d4e-40e1-8d59-f32700970597-sys\") pod \"node-exporter-2k242\" (UID: \"4137336d-3d4e-40e1-8d59-f32700970597\") " pod="openshift-monitoring/node-exporter-2k242" Nov 01 05:45:49.423668 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:49.423404 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4137336d-3d4e-40e1-8d59-f32700970597-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2k242\" (UID: \"4137336d-3d4e-40e1-8d59-f32700970597\") " pod="openshift-monitoring/node-exporter-2k242" Nov 01 05:45:49.423668 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:49.423515 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4137336d-3d4e-40e1-8d59-f32700970597-node-exporter-textfile\") pod \"node-exporter-2k242\" (UID: \"4137336d-3d4e-40e1-8d59-f32700970597\") " pod="openshift-monitoring/node-exporter-2k242" Nov 01 05:45:49.423668 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:49.423555 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4137336d-3d4e-40e1-8d59-f32700970597-node-exporter-wtmp\") pod \"node-exporter-2k242\" (UID: \"4137336d-3d4e-40e1-8d59-f32700970597\") " pod="openshift-monitoring/node-exporter-2k242" Nov 01 05:45:49.423828 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:49.423663 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4137336d-3d4e-40e1-8d59-f32700970597-node-exporter-accelerators-collector-config\") pod \"node-exporter-2k242\" (UID: \"4137336d-3d4e-40e1-8d59-f32700970597\") " pod="openshift-monitoring/node-exporter-2k242" Nov 01 05:45:49.423828 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:49.423710 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4137336d-3d4e-40e1-8d59-f32700970597-metrics-client-ca\") pod \"node-exporter-2k242\" (UID: \"4137336d-3d4e-40e1-8d59-f32700970597\") " pod="openshift-monitoring/node-exporter-2k242" Nov 01 05:45:49.423828 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:49.423769 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4137336d-3d4e-40e1-8d59-f32700970597-node-exporter-tls\") pod \"node-exporter-2k242\" (UID: \"4137336d-3d4e-40e1-8d59-f32700970597\") " pod="openshift-monitoring/node-exporter-2k242" Nov 01 05:45:49.423828 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:49.423808 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4137336d-3d4e-40e1-8d59-f32700970597-root\") pod \"node-exporter-2k242\" (UID: \"4137336d-3d4e-40e1-8d59-f32700970597\") " pod="openshift-monitoring/node-exporter-2k242" Nov 01 05:45:49.449850 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:49.449809 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 01 05:45:49.489119 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:49.489058 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 01 05:45:49.489119 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:49.489068 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-f427j\"" Nov 01 05:45:49.489119 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:49.489073 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 01 05:45:49.489119 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:49.489078 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 01 05:45:49.525178 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:49.525150 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4137336d-3d4e-40e1-8d59-f32700970597-node-exporter-accelerators-collector-config\") pod \"node-exporter-2k242\" (UID: \"4137336d-3d4e-40e1-8d59-f32700970597\") " pod="openshift-monitoring/node-exporter-2k242" Nov 01 05:45:49.525178 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:49.525183 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4137336d-3d4e-40e1-8d59-f32700970597-metrics-client-ca\") pod \"node-exporter-2k242\" (UID: \"4137336d-3d4e-40e1-8d59-f32700970597\") " pod="openshift-monitoring/node-exporter-2k242" Nov 01 05:45:49.525388 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:49.525201 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4137336d-3d4e-40e1-8d59-f32700970597-node-exporter-tls\") pod \"node-exporter-2k242\" (UID: \"4137336d-3d4e-40e1-8d59-f32700970597\") " pod="openshift-monitoring/node-exporter-2k242" Nov 01 05:45:49.525388 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:49.525228 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4137336d-3d4e-40e1-8d59-f32700970597-root\") pod \"node-exporter-2k242\" (UID: \"4137336d-3d4e-40e1-8d59-f32700970597\") " pod="openshift-monitoring/node-exporter-2k242" Nov 01 05:45:49.525388 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:49.525277 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-899zk\" (UniqueName: \"kubernetes.io/projected/4137336d-3d4e-40e1-8d59-f32700970597-kube-api-access-899zk\") pod \"node-exporter-2k242\" (UID: \"4137336d-3d4e-40e1-8d59-f32700970597\") " pod="openshift-monitoring/node-exporter-2k242" Nov 01 05:45:49.525388 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:49.525302 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4137336d-3d4e-40e1-8d59-f32700970597-sys\") pod \"node-exporter-2k242\" (UID: \"4137336d-3d4e-40e1-8d59-f32700970597\") " pod="openshift-monitoring/node-exporter-2k242" Nov 01 05:45:49.525388 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:49.525329 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4137336d-3d4e-40e1-8d59-f32700970597-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2k242\" (UID: \"4137336d-3d4e-40e1-8d59-f32700970597\") " pod="openshift-monitoring/node-exporter-2k242" Nov 01 05:45:49.525388 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:49.525358 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4137336d-3d4e-40e1-8d59-f32700970597-root\") pod \"node-exporter-2k242\" (UID: \"4137336d-3d4e-40e1-8d59-f32700970597\") " pod="openshift-monitoring/node-exporter-2k242" Nov 01 05:45:49.525731 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:49.525382 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4137336d-3d4e-40e1-8d59-f32700970597-sys\") pod \"node-exporter-2k242\" (UID: \"4137336d-3d4e-40e1-8d59-f32700970597\") " pod="openshift-monitoring/node-exporter-2k242" Nov 01 05:45:49.525731 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:49.525366 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4137336d-3d4e-40e1-8d59-f32700970597-node-exporter-textfile\") pod \"node-exporter-2k242\" (UID: \"4137336d-3d4e-40e1-8d59-f32700970597\") " pod="openshift-monitoring/node-exporter-2k242" Nov 01 05:45:49.525731 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:49.525457 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4137336d-3d4e-40e1-8d59-f32700970597-node-exporter-wtmp\") pod \"node-exporter-2k242\" (UID: \"4137336d-3d4e-40e1-8d59-f32700970597\") " pod="openshift-monitoring/node-exporter-2k242" Nov 01 05:45:49.525731 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:49.525681 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4137336d-3d4e-40e1-8d59-f32700970597-node-exporter-wtmp\") pod \"node-exporter-2k242\" (UID: \"4137336d-3d4e-40e1-8d59-f32700970597\") " pod="openshift-monitoring/node-exporter-2k242" Nov 01 05:45:49.525882 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:49.525755 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4137336d-3d4e-40e1-8d59-f32700970597-node-exporter-textfile\") pod \"node-exporter-2k242\" (UID: \"4137336d-3d4e-40e1-8d59-f32700970597\") " pod="openshift-monitoring/node-exporter-2k242" Nov 01 05:45:49.525970 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:49.525944 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4137336d-3d4e-40e1-8d59-f32700970597-metrics-client-ca\") pod \"node-exporter-2k242\" (UID: \"4137336d-3d4e-40e1-8d59-f32700970597\") " pod="openshift-monitoring/node-exporter-2k242" Nov 01 05:45:49.526081 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:49.525948 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4137336d-3d4e-40e1-8d59-f32700970597-node-exporter-accelerators-collector-config\") pod \"node-exporter-2k242\" (UID: \"4137336d-3d4e-40e1-8d59-f32700970597\") " pod="openshift-monitoring/node-exporter-2k242" Nov 01 05:45:49.527672 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:49.527654 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4137336d-3d4e-40e1-8d59-f32700970597-node-exporter-tls\") pod \"node-exporter-2k242\" (UID: \"4137336d-3d4e-40e1-8d59-f32700970597\") " pod="openshift-monitoring/node-exporter-2k242" Nov 01 05:45:49.527765 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:49.527745 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4137336d-3d4e-40e1-8d59-f32700970597-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2k242\" (UID: \"4137336d-3d4e-40e1-8d59-f32700970597\") " pod="openshift-monitoring/node-exporter-2k242" Nov 01 05:45:49.588423 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:49.588346 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-899zk\" (UniqueName: \"kubernetes.io/projected/4137336d-3d4e-40e1-8d59-f32700970597-kube-api-access-899zk\") pod \"node-exporter-2k242\" (UID: \"4137336d-3d4e-40e1-8d59-f32700970597\") " pod="openshift-monitoring/node-exporter-2k242" Nov 01 05:45:49.667084 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:49.667049 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2k242" Nov 01 05:45:49.698444 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:45:49.698419 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4137336d_3d4e_40e1_8d59_f32700970597.slice/crio-4bd92865fa35f299a09453e2c17a38d0b58fd357a643d61de48ba2b77ec0b096 WatchSource:0}: Error finding container 4bd92865fa35f299a09453e2c17a38d0b58fd357a643d61de48ba2b77ec0b096: Status 404 returned error can't find the container with id 4bd92865fa35f299a09453e2c17a38d0b58fd357a643d61de48ba2b77ec0b096 Nov 01 05:45:49.736137 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:49.736104 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2k242" event={"ID":"4137336d-3d4e-40e1-8d59-f32700970597","Type":"ContainerStarted","Data":"4bd92865fa35f299a09453e2c17a38d0b58fd357a643d61de48ba2b77ec0b096"} Nov 01 05:45:50.346323 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:50.346292 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:45:52.757541 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:52.757501 5040 generic.go:356] "Generic (PLEG): container finished" podID="4137336d-3d4e-40e1-8d59-f32700970597" containerID="dd28c60c33fb46dd1397373f374406ea78fca1e82b05188993554aa4bc7922e1" exitCode=0 Nov 01 05:45:52.757944 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:52.757558 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2k242" event={"ID":"4137336d-3d4e-40e1-8d59-f32700970597","Type":"ContainerDied","Data":"dd28c60c33fb46dd1397373f374406ea78fca1e82b05188993554aa4bc7922e1"} Nov 01 05:45:53.764185 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:53.764147 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2k242" event={"ID":"4137336d-3d4e-40e1-8d59-f32700970597","Type":"ContainerStarted","Data":"211a72e17927dcec44bafcd39c4ab0fb396a2ae48d3ebc820e73f40e79298f55"} Nov 01 05:45:53.764631 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:53.764204 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2k242" event={"ID":"4137336d-3d4e-40e1-8d59-f32700970597","Type":"ContainerStarted","Data":"76b55065fdf3a1a43e18a27eded4ec60e8a6c90de37095daacd7aea47045441f"} Nov 01 05:45:55.540155 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:55.540127 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ljcjx" Nov 01 05:45:55.657488 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:45:55.657422 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2k242" podStartSLOduration=4.574947097 podStartE2EDuration="6.657406416s" totalImagesPullingTime="2.008976172s" totalInitContainerRuntime="73.483147ms" isStatefulPod=true podCreationTimestamp="2026-11-01 05:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:53.887650016 +0000 UTC m=+32.542767186" watchObservedRunningTime="2026-11-01 05:45:55.657406416 +0000 UTC m=+34.312523639" Nov 01 05:46:03.645794 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:03.645757 5040 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n7mks_197fb18c-6703-4deb-93b3-0186d7ceed67/konnectivity-agent/0.log" limit=1024 Nov 01 05:46:03.895449 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:03.895411 5040 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n7mks_197fb18c-6703-4deb-93b3-0186d7ceed67/konnectivity-agent/0.log" limit=1024 Nov 01 05:46:04.061534 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:04.061471 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" podUID="f9ccbc38-43f9-4396-bb59-2f746b3d639d" containerName="registry" containerID="cri-o://b9edaa694b100ff6888b4eddd67c6018f975cab63b85b6a0a59e21431ea5e61d" gracePeriod=30 Nov 01 05:46:04.289307 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:04.289279 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:46:04.453184 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:04.453099 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9ccbc38-43f9-4396-bb59-2f746b3d639d-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f9ccbc38-43f9-4396-bb59-2f746b3d639d-trusted-ca\") pod \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " Nov 01 05:46:04.453184 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:04.453154 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9ccbc38-43f9-4396-bb59-2f746b3d639d-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f9ccbc38-43f9-4396-bb59-2f746b3d639d-registry-certificates\") pod \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " Nov 01 05:46:04.453416 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:04.453196 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9ccbc38-43f9-4396-bb59-2f746b3d639d-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f9ccbc38-43f9-4396-bb59-2f746b3d639d-ca-trust-extracted\") pod \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " Nov 01 05:46:04.453416 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:04.453257 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9ccbc38-43f9-4396-bb59-2f746b3d639d-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f9ccbc38-43f9-4396-bb59-2f746b3d639d-installation-pull-secrets\") pod \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " Nov 01 05:46:04.453416 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:04.453287 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-bound-sa-token\") pod \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " Nov 01 05:46:04.453416 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:04.453319 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-registry-tls\" (UniqueName: \"kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-registry-tls\") pod \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " Nov 01 05:46:04.453416 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:04.453352 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-kube-api-access-vqqbs\" (UniqueName: \"kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-kube-api-access-vqqbs\") pod \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " Nov 01 05:46:04.453416 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:04.453391 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9ccbc38-43f9-4396-bb59-2f746b3d639d-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f9ccbc38-43f9-4396-bb59-2f746b3d639d-image-registry-private-configuration\") pod \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\" (UID: \"f9ccbc38-43f9-4396-bb59-2f746b3d639d\") " Nov 01 05:46:04.453737 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:04.453667 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9ccbc38-43f9-4396-bb59-2f746b3d639d-trusted-ca" pod "f9ccbc38-43f9-4396-bb59-2f746b3d639d" (UID: "f9ccbc38-43f9-4396-bb59-2f746b3d639d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:46:04.453804 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:04.453726 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9ccbc38-43f9-4396-bb59-2f746b3d639d-registry-certificates" pod "f9ccbc38-43f9-4396-bb59-2f746b3d639d" (UID: "f9ccbc38-43f9-4396-bb59-2f746b3d639d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:46:04.455969 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:04.455934 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-bound-sa-token" pod "f9ccbc38-43f9-4396-bb59-2f746b3d639d" (UID: "f9ccbc38-43f9-4396-bb59-2f746b3d639d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:46:04.456105 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:04.456010 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9ccbc38-43f9-4396-bb59-2f746b3d639d-installation-pull-secrets" pod "f9ccbc38-43f9-4396-bb59-2f746b3d639d" (UID: "f9ccbc38-43f9-4396-bb59-2f746b3d639d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:46:04.456105 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:04.456089 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9ccbc38-43f9-4396-bb59-2f746b3d639d-image-registry-private-configuration" pod "f9ccbc38-43f9-4396-bb59-2f746b3d639d" (UID: "f9ccbc38-43f9-4396-bb59-2f746b3d639d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:46:04.456193 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:04.456169 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-registry-tls" pod "f9ccbc38-43f9-4396-bb59-2f746b3d639d" (UID: "f9ccbc38-43f9-4396-bb59-2f746b3d639d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:46:04.456224 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:04.456210 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-kube-api-access-vqqbs" pod "f9ccbc38-43f9-4396-bb59-2f746b3d639d" (UID: "f9ccbc38-43f9-4396-bb59-2f746b3d639d"). InnerVolumeSpecName "kube-api-access-vqqbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:46:04.460810 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:04.460739 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9ccbc38-43f9-4396-bb59-2f746b3d639d-ca-trust-extracted" pod "f9ccbc38-43f9-4396-bb59-2f746b3d639d" (UID: "f9ccbc38-43f9-4396-bb59-2f746b3d639d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:46:04.554710 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:04.554672 5040 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f9ccbc38-43f9-4396-bb59-2f746b3d639d-image-registry-private-configuration\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:04.554710 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:04.554704 5040 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f9ccbc38-43f9-4396-bb59-2f746b3d639d-trusted-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:04.554710 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:04.554720 5040 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f9ccbc38-43f9-4396-bb59-2f746b3d639d-registry-certificates\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:04.554949 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:04.554733 5040 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f9ccbc38-43f9-4396-bb59-2f746b3d639d-ca-trust-extracted\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:04.554949 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:04.554745 5040 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f9ccbc38-43f9-4396-bb59-2f746b3d639d-installation-pull-secrets\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:04.554949 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:04.554757 5040 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-bound-sa-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:04.554949 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:04.554769 5040 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-registry-tls\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:04.554949 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:04.554782 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqqbs\" (UniqueName: \"kubernetes.io/projected/f9ccbc38-43f9-4396-bb59-2f746b3d639d-kube-api-access-vqqbs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:04.807897 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:04.807863 5040 generic.go:356] "Generic (PLEG): container finished" podID="f9ccbc38-43f9-4396-bb59-2f746b3d639d" containerID="b9edaa694b100ff6888b4eddd67c6018f975cab63b85b6a0a59e21431ea5e61d" exitCode=0 Nov 01 05:46:04.808351 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:04.807916 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" Nov 01 05:46:04.808351 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:04.807946 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" event={"ID":"f9ccbc38-43f9-4396-bb59-2f746b3d639d","Type":"ContainerDied","Data":"b9edaa694b100ff6888b4eddd67c6018f975cab63b85b6a0a59e21431ea5e61d"} Nov 01 05:46:04.808351 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:04.808010 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f55979cbb-ngg8z" event={"ID":"f9ccbc38-43f9-4396-bb59-2f746b3d639d","Type":"ContainerDied","Data":"006608d8bd74a5415ebf37346ec8b061ea51019dcbdcc3df8eb2525c696efcc5"} Nov 01 05:46:04.808351 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:04.808040 5040 scope.go:122] "RemoveContainer" containerID="b9edaa694b100ff6888b4eddd67c6018f975cab63b85b6a0a59e21431ea5e61d" Nov 01 05:46:04.817176 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:04.817161 5040 scope.go:122] "RemoveContainer" containerID="b9edaa694b100ff6888b4eddd67c6018f975cab63b85b6a0a59e21431ea5e61d" Nov 01 05:46:04.817456 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:46:04.817436 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9edaa694b100ff6888b4eddd67c6018f975cab63b85b6a0a59e21431ea5e61d\": container with ID starting with b9edaa694b100ff6888b4eddd67c6018f975cab63b85b6a0a59e21431ea5e61d not found: ID does not exist" containerID="b9edaa694b100ff6888b4eddd67c6018f975cab63b85b6a0a59e21431ea5e61d" Nov 01 05:46:04.817552 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:04.817464 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9edaa694b100ff6888b4eddd67c6018f975cab63b85b6a0a59e21431ea5e61d"} err="failed to get container status \"b9edaa694b100ff6888b4eddd67c6018f975cab63b85b6a0a59e21431ea5e61d\": rpc error: code = NotFound desc = could not find container \"b9edaa694b100ff6888b4eddd67c6018f975cab63b85b6a0a59e21431ea5e61d\": container with ID starting with b9edaa694b100ff6888b4eddd67c6018f975cab63b85b6a0a59e21431ea5e61d not found: ID does not exist" Nov 01 05:46:04.973630 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:04.973601 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f55979cbb-ngg8z"] Nov 01 05:46:05.024469 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:05.024437 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5f55979cbb-ngg8z"] Nov 01 05:46:05.512687 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:05.512654 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9ccbc38-43f9-4396-bb59-2f746b3d639d" path="/var/lib/kubelet/pods/f9ccbc38-43f9-4396-bb59-2f746b3d639d/volumes" Nov 01 05:46:09.974014 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:46:09.973972 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 05:46:12.044057 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:12.044022 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-847d56945f-lt2tl"] Nov 01 05:46:37.063953 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:37.063907 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-847d56945f-lt2tl" podUID="7729d415-70e5-4be3-a9cc-bbd13e065c71" containerName="registry" containerID="cri-o://246378f47402702a056c8ddc696564bf27ccfe280249c02d5e7b95008b58f5c6" gracePeriod=30 Nov 01 05:46:37.310590 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:37.310566 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:46:37.430395 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:37.430317 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7729d415-70e5-4be3-a9cc-bbd13e065c71-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7729d415-70e5-4be3-a9cc-bbd13e065c71-trusted-ca\") pod \"7729d415-70e5-4be3-a9cc-bbd13e065c71\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " Nov 01 05:46:37.430395 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:37.430361 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-kube-api-access-72qwb\" (UniqueName: \"kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-kube-api-access-72qwb\") pod \"7729d415-70e5-4be3-a9cc-bbd13e065c71\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " Nov 01 05:46:37.430395 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:37.430389 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-bound-sa-token\") pod \"7729d415-70e5-4be3-a9cc-bbd13e065c71\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " Nov 01 05:46:37.430667 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:37.430409 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7729d415-70e5-4be3-a9cc-bbd13e065c71-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7729d415-70e5-4be3-a9cc-bbd13e065c71-registry-certificates\") pod \"7729d415-70e5-4be3-a9cc-bbd13e065c71\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " Nov 01 05:46:37.430667 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:37.430441 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-registry-tls\" (UniqueName: \"kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-registry-tls\") pod \"7729d415-70e5-4be3-a9cc-bbd13e065c71\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " Nov 01 05:46:37.430667 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:37.430503 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7729d415-70e5-4be3-a9cc-bbd13e065c71-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7729d415-70e5-4be3-a9cc-bbd13e065c71-image-registry-private-configuration\") pod \"7729d415-70e5-4be3-a9cc-bbd13e065c71\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " Nov 01 05:46:37.430667 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:37.430540 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7729d415-70e5-4be3-a9cc-bbd13e065c71-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7729d415-70e5-4be3-a9cc-bbd13e065c71-ca-trust-extracted\") pod \"7729d415-70e5-4be3-a9cc-bbd13e065c71\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " Nov 01 05:46:37.430667 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:37.430579 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7729d415-70e5-4be3-a9cc-bbd13e065c71-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7729d415-70e5-4be3-a9cc-bbd13e065c71-installation-pull-secrets\") pod \"7729d415-70e5-4be3-a9cc-bbd13e065c71\" (UID: \"7729d415-70e5-4be3-a9cc-bbd13e065c71\") " Nov 01 05:46:37.430922 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:37.430859 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7729d415-70e5-4be3-a9cc-bbd13e065c71-registry-certificates" pod "7729d415-70e5-4be3-a9cc-bbd13e065c71" (UID: "7729d415-70e5-4be3-a9cc-bbd13e065c71"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:46:37.431419 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:37.431388 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7729d415-70e5-4be3-a9cc-bbd13e065c71-trusted-ca" pod "7729d415-70e5-4be3-a9cc-bbd13e065c71" (UID: "7729d415-70e5-4be3-a9cc-bbd13e065c71"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:46:37.433029 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:37.432964 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7729d415-70e5-4be3-a9cc-bbd13e065c71-image-registry-private-configuration" pod "7729d415-70e5-4be3-a9cc-bbd13e065c71" (UID: "7729d415-70e5-4be3-a9cc-bbd13e065c71"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:46:37.433156 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:37.433130 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7729d415-70e5-4be3-a9cc-bbd13e065c71-installation-pull-secrets" pod "7729d415-70e5-4be3-a9cc-bbd13e065c71" (UID: "7729d415-70e5-4be3-a9cc-bbd13e065c71"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:46:37.433251 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:37.433232 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-bound-sa-token" pod "7729d415-70e5-4be3-a9cc-bbd13e065c71" (UID: "7729d415-70e5-4be3-a9cc-bbd13e065c71"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:46:37.433389 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:37.433365 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-registry-tls" pod "7729d415-70e5-4be3-a9cc-bbd13e065c71" (UID: "7729d415-70e5-4be3-a9cc-bbd13e065c71"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:46:37.434706 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:37.434689 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-kube-api-access-72qwb" pod "7729d415-70e5-4be3-a9cc-bbd13e065c71" (UID: "7729d415-70e5-4be3-a9cc-bbd13e065c71"). InnerVolumeSpecName "kube-api-access-72qwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:46:37.437829 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:37.437805 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7729d415-70e5-4be3-a9cc-bbd13e065c71-ca-trust-extracted" pod "7729d415-70e5-4be3-a9cc-bbd13e065c71" (UID: "7729d415-70e5-4be3-a9cc-bbd13e065c71"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:46:37.532052 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:37.532026 5040 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7729d415-70e5-4be3-a9cc-bbd13e065c71-image-registry-private-configuration\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:37.532052 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:37.532051 5040 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7729d415-70e5-4be3-a9cc-bbd13e065c71-ca-trust-extracted\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:37.532207 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:37.532062 5040 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7729d415-70e5-4be3-a9cc-bbd13e065c71-installation-pull-secrets\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:37.532207 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:37.532073 5040 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7729d415-70e5-4be3-a9cc-bbd13e065c71-trusted-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:37.532207 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:37.532082 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72qwb\" (UniqueName: \"kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-kube-api-access-72qwb\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:37.532207 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:37.532091 5040 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-bound-sa-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:37.532207 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:37.532100 5040 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7729d415-70e5-4be3-a9cc-bbd13e065c71-registry-certificates\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:37.532207 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:37.532109 5040 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7729d415-70e5-4be3-a9cc-bbd13e065c71-registry-tls\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:37.940497 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:37.940443 5040 generic.go:356] "Generic (PLEG): container finished" podID="7729d415-70e5-4be3-a9cc-bbd13e065c71" containerID="246378f47402702a056c8ddc696564bf27ccfe280249c02d5e7b95008b58f5c6" exitCode=0 Nov 01 05:46:37.940685 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:37.940520 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-847d56945f-lt2tl" Nov 01 05:46:37.940685 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:37.940520 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-847d56945f-lt2tl" event={"ID":"7729d415-70e5-4be3-a9cc-bbd13e065c71","Type":"ContainerDied","Data":"246378f47402702a056c8ddc696564bf27ccfe280249c02d5e7b95008b58f5c6"} Nov 01 05:46:37.940685 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:37.940609 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-847d56945f-lt2tl" event={"ID":"7729d415-70e5-4be3-a9cc-bbd13e065c71","Type":"ContainerDied","Data":"6d613cf316fe8124c019364ee2e2e8ce200e854e8ba730eb54c48c6553d622b2"} Nov 01 05:46:37.940685 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:37.940637 5040 scope.go:122] "RemoveContainer" containerID="246378f47402702a056c8ddc696564bf27ccfe280249c02d5e7b95008b58f5c6" Nov 01 05:46:37.948576 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:37.948555 5040 scope.go:122] "RemoveContainer" containerID="246378f47402702a056c8ddc696564bf27ccfe280249c02d5e7b95008b58f5c6" Nov 01 05:46:37.948834 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:46:37.948814 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"246378f47402702a056c8ddc696564bf27ccfe280249c02d5e7b95008b58f5c6\": container with ID starting with 246378f47402702a056c8ddc696564bf27ccfe280249c02d5e7b95008b58f5c6 not found: ID does not exist" containerID="246378f47402702a056c8ddc696564bf27ccfe280249c02d5e7b95008b58f5c6" Nov 01 05:46:37.948887 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:37.948847 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"246378f47402702a056c8ddc696564bf27ccfe280249c02d5e7b95008b58f5c6"} err="failed to get container status \"246378f47402702a056c8ddc696564bf27ccfe280249c02d5e7b95008b58f5c6\": rpc error: code = NotFound desc = could not find container \"246378f47402702a056c8ddc696564bf27ccfe280249c02d5e7b95008b58f5c6\": container with ID starting with 246378f47402702a056c8ddc696564bf27ccfe280249c02d5e7b95008b58f5c6 not found: ID does not exist" Nov 01 05:46:38.107409 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:38.107378 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-847d56945f-lt2tl"] Nov 01 05:46:38.158259 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:38.158230 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-847d56945f-lt2tl"] Nov 01 05:46:39.514150 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:39.514118 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7729d415-70e5-4be3-a9cc-bbd13e065c71" path="/var/lib/kubelet/pods/7729d415-70e5-4be3-a9cc-bbd13e065c71/volumes" Nov 01 05:46:39.971102 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:46:39.971022 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 05:46:42.962824 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:42.962789 5040 generic.go:356] "Generic (PLEG): container finished" podID="293a6ef2-f0c9-4390-ac82-028792a3b64d" containerID="f2cd7eb153d1b6552f6b136b4f34d5e2af55d9f5f2050685027b0917d10e10a3" exitCode=0 Nov 01 05:46:42.963206 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:42.962841 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vr978" event={"ID":"293a6ef2-f0c9-4390-ac82-028792a3b64d","Type":"ContainerDied","Data":"f2cd7eb153d1b6552f6b136b4f34d5e2af55d9f5f2050685027b0917d10e10a3"} Nov 01 05:46:42.963206 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:42.963171 5040 scope.go:122] "RemoveContainer" containerID="f2cd7eb153d1b6552f6b136b4f34d5e2af55d9f5f2050685027b0917d10e10a3" Nov 01 05:46:43.968569 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:43.968528 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vr978" event={"ID":"293a6ef2-f0c9-4390-ac82-028792a3b64d","Type":"ContainerStarted","Data":"88e21acb9b629dccf941f3fa872a82472ffe5ec3d51ec303d7dfaac9a4853a37"} Nov 01 05:46:43.969965 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:43.969940 5040 generic.go:356] "Generic (PLEG): container finished" podID="47088d4b-2aa6-4f69-9847-f361e4fa5f7a" containerID="5cfa7ce8914b360f1d260db3a06d3c32161a7702d2da0c621a7fc7c945bc0900" exitCode=0 Nov 01 05:46:43.970104 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:43.970007 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-nh7q2-cs6dz" event={"ID":"47088d4b-2aa6-4f69-9847-f361e4fa5f7a","Type":"ContainerDied","Data":"5cfa7ce8914b360f1d260db3a06d3c32161a7702d2da0c621a7fc7c945bc0900"} Nov 01 05:46:45.119857 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:45.119833 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-nh7q2-cs6dz" Nov 01 05:46:45.195208 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:45.195179 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-archives-path\") pod \"47088d4b-2aa6-4f69-9847-f361e4fa5f7a\" (UID: \"47088d4b-2aa6-4f69-9847-f361e4fa5f7a\") " Nov 01 05:46:45.195208 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:45.195215 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-trusted-ca-bundle\") pod \"47088d4b-2aa6-4f69-9847-f361e4fa5f7a\" (UID: \"47088d4b-2aa6-4f69-9847-f361e4fa5f7a\") " Nov 01 05:46:45.195402 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:45.195241 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-service-ca-bundle\") pod \"47088d4b-2aa6-4f69-9847-f361e4fa5f7a\" (UID: \"47088d4b-2aa6-4f69-9847-f361e4fa5f7a\") " Nov 01 05:46:45.195402 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:45.195332 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-kube-api-access-5gcvx\" (UniqueName: \"kubernetes.io/projected/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-kube-api-access-5gcvx\") pod \"47088d4b-2aa6-4f69-9847-f361e4fa5f7a\" (UID: \"47088d4b-2aa6-4f69-9847-f361e4fa5f7a\") " Nov 01 05:46:45.195592 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:45.195566 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-archives-path" pod "47088d4b-2aa6-4f69-9847-f361e4fa5f7a" (UID: "47088d4b-2aa6-4f69-9847-f361e4fa5f7a"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:46:45.195691 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:45.195667 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-service-ca-bundle" pod "47088d4b-2aa6-4f69-9847-f361e4fa5f7a" (UID: "47088d4b-2aa6-4f69-9847-f361e4fa5f7a"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:46:45.195743 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:45.195672 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-trusted-ca-bundle" pod "47088d4b-2aa6-4f69-9847-f361e4fa5f7a" (UID: "47088d4b-2aa6-4f69-9847-f361e4fa5f7a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:46:45.197553 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:45.197530 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-kube-api-access-5gcvx" pod "47088d4b-2aa6-4f69-9847-f361e4fa5f7a" (UID: "47088d4b-2aa6-4f69-9847-f361e4fa5f7a"). InnerVolumeSpecName "kube-api-access-5gcvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:46:45.296408 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:45.296365 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gcvx\" (UniqueName: \"kubernetes.io/projected/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-kube-api-access-5gcvx\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:45.296408 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:45.296400 5040 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-archives-path\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:45.296408 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:45.296411 5040 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-trusted-ca-bundle\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:45.296675 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:45.296421 5040 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47088d4b-2aa6-4f69-9847-f361e4fa5f7a-service-ca-bundle\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:45.979359 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:45.979322 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-nh7q2-cs6dz" event={"ID":"47088d4b-2aa6-4f69-9847-f361e4fa5f7a","Type":"ContainerDied","Data":"a9dd32ace74afdc7105a4060ccacc954ea3c04bed453244c8496e0745d835782"} Nov 01 05:46:45.979359 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:45.979348 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-nh7q2-cs6dz" Nov 01 05:46:45.979359 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:45.979367 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9dd32ace74afdc7105a4060ccacc954ea3c04bed453244c8496e0745d835782" Nov 01 05:46:47.988137 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:47.988100 5040 generic.go:356] "Generic (PLEG): container finished" podID="109327c0-f8f8-4dde-9cfe-489bf5a7eaad" containerID="32e2eca69bde1a4d7389195dfa332e6f9eafb097537c3b0f7f952937209a5729" exitCode=0 Nov 01 05:46:47.988529 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:47.988174 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95sds" event={"ID":"109327c0-f8f8-4dde-9cfe-489bf5a7eaad","Type":"ContainerDied","Data":"32e2eca69bde1a4d7389195dfa332e6f9eafb097537c3b0f7f952937209a5729"} Nov 01 05:46:47.988601 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:47.988535 5040 scope.go:122] "RemoveContainer" containerID="32e2eca69bde1a4d7389195dfa332e6f9eafb097537c3b0f7f952937209a5729" Nov 01 05:46:48.993542 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:48.993508 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95sds" event={"ID":"109327c0-f8f8-4dde-9cfe-489bf5a7eaad","Type":"ContainerStarted","Data":"9eb7205d3623d00b2295e2e116115e5d5a87feff507def240f3d1ac318859025"} Nov 01 05:46:57.030217 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:57.030184 5040 generic.go:356] "Generic (PLEG): container finished" podID="cbc9c127-21a0-4020-a093-d02b4061db5a" containerID="a9909954820f8641bc7ca4ec520502b8f4d259fb20dd5dcb8f777b08b24c49f5" exitCode=0 Nov 01 05:46:57.030626 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:57.030248 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-r8n4b" event={"ID":"cbc9c127-21a0-4020-a093-d02b4061db5a","Type":"ContainerDied","Data":"a9909954820f8641bc7ca4ec520502b8f4d259fb20dd5dcb8f777b08b24c49f5"} Nov 01 05:46:57.030626 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:57.030572 5040 scope.go:122] "RemoveContainer" containerID="a9909954820f8641bc7ca4ec520502b8f4d259fb20dd5dcb8f777b08b24c49f5" Nov 01 05:46:58.041747 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:46:58.041701 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-r8n4b" event={"ID":"cbc9c127-21a0-4020-a093-d02b4061db5a","Type":"ContainerStarted","Data":"6ca4f43664acd6854e64e7bd088d0a6691b0a025b5fa193ca72fd0030cc14147"} Nov 01 05:47:09.971521 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:47:09.971466 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 05:47:39.971507 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:47:39.971410 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 05:48:09.971579 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:48:09.971542 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 05:48:39.971788 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:48:39.971746 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 05:49:09.971359 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:49:09.971320 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 05:49:39.971587 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:49:39.971549 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 05:50:09.971376 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:50:09.971340 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 05:50:21.519748 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:21.519721 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 05:50:21.526365 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:21.526334 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 05:50:30.565348 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:30.565310 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644scwvg"] Nov 01 05:50:30.565776 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:30.565640 5040 cpu_manager.go:414] "containerMap: removing container" podUID="f9ccbc38-43f9-4396-bb59-2f746b3d639d" containerName="registry" Nov 01 05:50:30.565776 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:30.565651 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9ccbc38-43f9-4396-bb59-2f746b3d639d" containerName="registry" Nov 01 05:50:30.565776 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:30.565686 5040 cpu_manager.go:414] "containerMap: removing container" podUID="47088d4b-2aa6-4f69-9847-f361e4fa5f7a" containerName="insights-gathering" Nov 01 05:50:30.565776 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:30.565691 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47088d4b-2aa6-4f69-9847-f361e4fa5f7a" containerName="insights-gathering" Nov 01 05:50:30.565776 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:30.565699 5040 cpu_manager.go:414] "containerMap: removing container" podUID="7729d415-70e5-4be3-a9cc-bbd13e065c71" containerName="registry" Nov 01 05:50:30.565776 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:30.565704 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7729d415-70e5-4be3-a9cc-bbd13e065c71" containerName="registry" Nov 01 05:50:30.565776 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:30.565757 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="47088d4b-2aa6-4f69-9847-f361e4fa5f7a" containerName="insights-gathering" Nov 01 05:50:30.565776 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:30.565765 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="7729d415-70e5-4be3-a9cc-bbd13e065c71" containerName="registry" Nov 01 05:50:30.565776 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:30.565773 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9ccbc38-43f9-4396-bb59-2f746b3d639d" containerName="registry" Nov 01 05:50:30.569065 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:30.569045 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644scwvg" Nov 01 05:50:30.622285 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:30.622212 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 01 05:50:30.649630 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:30.649593 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e539788-0929-469e-9bc0-97cd39704834-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644scwvg\" (UID: \"2e539788-0929-469e-9bc0-97cd39704834\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644scwvg" Nov 01 05:50:30.649789 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:30.649629 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e539788-0929-469e-9bc0-97cd39704834-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644scwvg\" (UID: \"2e539788-0929-469e-9bc0-97cd39704834\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644scwvg" Nov 01 05:50:30.649789 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:30.649738 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhbr7\" (UniqueName: \"kubernetes.io/projected/2e539788-0929-469e-9bc0-97cd39704834-kube-api-access-mhbr7\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644scwvg\" (UID: \"2e539788-0929-469e-9bc0-97cd39704834\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644scwvg" Nov 01 05:50:30.672941 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:30.672910 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 01 05:50:30.673193 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:30.673173 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-8f25r\"" Nov 01 05:50:30.674189 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:30.674171 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644scwvg"] Nov 01 05:50:30.751055 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:30.751016 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e539788-0929-469e-9bc0-97cd39704834-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644scwvg\" (UID: \"2e539788-0929-469e-9bc0-97cd39704834\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644scwvg" Nov 01 05:50:30.751055 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:30.751053 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e539788-0929-469e-9bc0-97cd39704834-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644scwvg\" (UID: \"2e539788-0929-469e-9bc0-97cd39704834\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644scwvg" Nov 01 05:50:30.751257 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:30.751104 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhbr7\" (UniqueName: \"kubernetes.io/projected/2e539788-0929-469e-9bc0-97cd39704834-kube-api-access-mhbr7\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644scwvg\" (UID: \"2e539788-0929-469e-9bc0-97cd39704834\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644scwvg" Nov 01 05:50:30.751542 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:30.751520 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e539788-0929-469e-9bc0-97cd39704834-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644scwvg\" (UID: \"2e539788-0929-469e-9bc0-97cd39704834\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644scwvg" Nov 01 05:50:30.751582 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:30.751529 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e539788-0929-469e-9bc0-97cd39704834-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644scwvg\" (UID: \"2e539788-0929-469e-9bc0-97cd39704834\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644scwvg" Nov 01 05:50:30.813282 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:30.813251 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhbr7\" (UniqueName: \"kubernetes.io/projected/2e539788-0929-469e-9bc0-97cd39704834-kube-api-access-mhbr7\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644scwvg\" (UID: \"2e539788-0929-469e-9bc0-97cd39704834\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644scwvg" Nov 01 05:50:30.884832 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:30.884750 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644scwvg" Nov 01 05:50:31.254742 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:31.254715 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644scwvg"] Nov 01 05:50:31.257292 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:50:31.257266 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e539788_0929_469e_9bc0_97cd39704834.slice/crio-313e992b34fddf2026c78d0a3cd8a8dd52d6e65a463477f1a48eef81d3444855 WatchSource:0}: Error finding container 313e992b34fddf2026c78d0a3cd8a8dd52d6e65a463477f1a48eef81d3444855: Status 404 returned error can't find the container with id 313e992b34fddf2026c78d0a3cd8a8dd52d6e65a463477f1a48eef81d3444855 Nov 01 05:50:31.259285 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:31.259266 5040 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 05:50:31.871108 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:31.871076 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644scwvg" event={"ID":"2e539788-0929-469e-9bc0-97cd39704834","Type":"ContainerStarted","Data":"313e992b34fddf2026c78d0a3cd8a8dd52d6e65a463477f1a48eef81d3444855"} Nov 01 05:50:33.181674 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.181638 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gg2m2"] Nov 01 05:50:33.185245 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.185225 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gg2m2" Nov 01 05:50:33.272750 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.272714 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1eedcd2c-e8a2-4244-a764-bb19b5aa726f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gg2m2\" (UID: \"1eedcd2c-e8a2-4244-a764-bb19b5aa726f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gg2m2" Nov 01 05:50:33.272922 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.272810 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5d2r\" (UniqueName: \"kubernetes.io/projected/1eedcd2c-e8a2-4244-a764-bb19b5aa726f-kube-api-access-k5d2r\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gg2m2\" (UID: \"1eedcd2c-e8a2-4244-a764-bb19b5aa726f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gg2m2" Nov 01 05:50:33.272922 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.272873 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1eedcd2c-e8a2-4244-a764-bb19b5aa726f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gg2m2\" (UID: \"1eedcd2c-e8a2-4244-a764-bb19b5aa726f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gg2m2" Nov 01 05:50:33.289428 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.289401 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-q528d\"" Nov 01 05:50:33.290942 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.290916 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gg2m2"] Nov 01 05:50:33.292243 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.292215 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c92d2r"] Nov 01 05:50:33.295774 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.295756 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c92d2r" Nov 01 05:50:33.349175 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.349137 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-r9h67\"" Nov 01 05:50:33.355157 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.355129 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c92d2r"] Nov 01 05:50:33.373764 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.373729 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1eedcd2c-e8a2-4244-a764-bb19b5aa726f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gg2m2\" (UID: \"1eedcd2c-e8a2-4244-a764-bb19b5aa726f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gg2m2" Nov 01 05:50:33.373914 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.373798 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqx4d\" (UniqueName: \"kubernetes.io/projected/4bd01e10-2e4a-4d5f-abda-ea50f28586b8-kube-api-access-tqx4d\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c92d2r\" (UID: \"4bd01e10-2e4a-4d5f-abda-ea50f28586b8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c92d2r" Nov 01 05:50:33.373914 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.373856 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4bd01e10-2e4a-4d5f-abda-ea50f28586b8-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c92d2r\" (UID: \"4bd01e10-2e4a-4d5f-abda-ea50f28586b8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c92d2r" Nov 01 05:50:33.373914 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.373889 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4bd01e10-2e4a-4d5f-abda-ea50f28586b8-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c92d2r\" (UID: \"4bd01e10-2e4a-4d5f-abda-ea50f28586b8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c92d2r" Nov 01 05:50:33.374217 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.373984 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5d2r\" (UniqueName: \"kubernetes.io/projected/1eedcd2c-e8a2-4244-a764-bb19b5aa726f-kube-api-access-k5d2r\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gg2m2\" (UID: \"1eedcd2c-e8a2-4244-a764-bb19b5aa726f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gg2m2" Nov 01 05:50:33.374217 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.374165 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1eedcd2c-e8a2-4244-a764-bb19b5aa726f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gg2m2\" (UID: \"1eedcd2c-e8a2-4244-a764-bb19b5aa726f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gg2m2" Nov 01 05:50:33.374322 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.374236 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1eedcd2c-e8a2-4244-a764-bb19b5aa726f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gg2m2\" (UID: \"1eedcd2c-e8a2-4244-a764-bb19b5aa726f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gg2m2" Nov 01 05:50:33.374538 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.374520 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1eedcd2c-e8a2-4244-a764-bb19b5aa726f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gg2m2\" (UID: \"1eedcd2c-e8a2-4244-a764-bb19b5aa726f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gg2m2" Nov 01 05:50:33.386977 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.386952 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atmqq6"] Nov 01 05:50:33.390491 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.390466 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atmqq6" Nov 01 05:50:33.438005 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.437936 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5d2r\" (UniqueName: \"kubernetes.io/projected/1eedcd2c-e8a2-4244-a764-bb19b5aa726f-kube-api-access-k5d2r\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gg2m2\" (UID: \"1eedcd2c-e8a2-4244-a764-bb19b5aa726f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gg2m2" Nov 01 05:50:33.445943 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.445914 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-wrn2b\"" Nov 01 05:50:33.450649 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.450620 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atmqq6"] Nov 01 05:50:33.475640 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.475614 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqx4d\" (UniqueName: \"kubernetes.io/projected/4bd01e10-2e4a-4d5f-abda-ea50f28586b8-kube-api-access-tqx4d\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c92d2r\" (UID: \"4bd01e10-2e4a-4d5f-abda-ea50f28586b8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c92d2r" Nov 01 05:50:33.475805 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.475648 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13895149-4860-44a1-9ecb-9d054efc3aa2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atmqq6\" (UID: \"13895149-4860-44a1-9ecb-9d054efc3aa2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atmqq6" Nov 01 05:50:33.475805 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.475685 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hshw\" (UniqueName: \"kubernetes.io/projected/13895149-4860-44a1-9ecb-9d054efc3aa2-kube-api-access-9hshw\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atmqq6\" (UID: \"13895149-4860-44a1-9ecb-9d054efc3aa2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atmqq6" Nov 01 05:50:33.475805 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.475718 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4bd01e10-2e4a-4d5f-abda-ea50f28586b8-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c92d2r\" (UID: \"4bd01e10-2e4a-4d5f-abda-ea50f28586b8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c92d2r" Nov 01 05:50:33.475805 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.475743 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4bd01e10-2e4a-4d5f-abda-ea50f28586b8-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c92d2r\" (UID: \"4bd01e10-2e4a-4d5f-abda-ea50f28586b8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c92d2r" Nov 01 05:50:33.476041 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.475833 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13895149-4860-44a1-9ecb-9d054efc3aa2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atmqq6\" (UID: \"13895149-4860-44a1-9ecb-9d054efc3aa2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atmqq6" Nov 01 05:50:33.476222 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.476133 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4bd01e10-2e4a-4d5f-abda-ea50f28586b8-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c92d2r\" (UID: \"4bd01e10-2e4a-4d5f-abda-ea50f28586b8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c92d2r" Nov 01 05:50:33.476222 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.476181 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4bd01e10-2e4a-4d5f-abda-ea50f28586b8-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c92d2r\" (UID: \"4bd01e10-2e4a-4d5f-abda-ea50f28586b8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c92d2r" Nov 01 05:50:33.487726 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.487665 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czbwm"] Nov 01 05:50:33.491896 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.491875 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czbwm" Nov 01 05:50:33.502904 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.502875 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gg2m2" Nov 01 05:50:33.540433 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.540400 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqx4d\" (UniqueName: \"kubernetes.io/projected/4bd01e10-2e4a-4d5f-abda-ea50f28586b8-kube-api-access-tqx4d\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c92d2r\" (UID: \"4bd01e10-2e4a-4d5f-abda-ea50f28586b8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c92d2r" Nov 01 05:50:33.548725 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.548696 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-8nk6q\"" Nov 01 05:50:33.550174 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.550098 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czbwm"] Nov 01 05:50:33.577026 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.576964 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5886aeb7-6e38-435a-9bd8-acb6f21ddc70-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czbwm\" (UID: \"5886aeb7-6e38-435a-9bd8-acb6f21ddc70\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czbwm" Nov 01 05:50:33.577026 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.577003 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13895149-4860-44a1-9ecb-9d054efc3aa2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atmqq6\" (UID: \"13895149-4860-44a1-9ecb-9d054efc3aa2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atmqq6" Nov 01 05:50:33.577026 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.577026 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5886aeb7-6e38-435a-9bd8-acb6f21ddc70-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czbwm\" (UID: \"5886aeb7-6e38-435a-9bd8-acb6f21ddc70\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czbwm" Nov 01 05:50:33.577206 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.577058 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13895149-4860-44a1-9ecb-9d054efc3aa2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atmqq6\" (UID: \"13895149-4860-44a1-9ecb-9d054efc3aa2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atmqq6" Nov 01 05:50:33.577206 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.577074 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jhwn\" (UniqueName: \"kubernetes.io/projected/5886aeb7-6e38-435a-9bd8-acb6f21ddc70-kube-api-access-2jhwn\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czbwm\" (UID: \"5886aeb7-6e38-435a-9bd8-acb6f21ddc70\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czbwm" Nov 01 05:50:33.577206 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.577098 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hshw\" (UniqueName: \"kubernetes.io/projected/13895149-4860-44a1-9ecb-9d054efc3aa2-kube-api-access-9hshw\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atmqq6\" (UID: \"13895149-4860-44a1-9ecb-9d054efc3aa2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atmqq6" Nov 01 05:50:33.577712 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.577656 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13895149-4860-44a1-9ecb-9d054efc3aa2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atmqq6\" (UID: \"13895149-4860-44a1-9ecb-9d054efc3aa2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atmqq6" Nov 01 05:50:33.577712 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.577682 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13895149-4860-44a1-9ecb-9d054efc3aa2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atmqq6\" (UID: \"13895149-4860-44a1-9ecb-9d054efc3aa2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atmqq6" Nov 01 05:50:33.613285 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.613261 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c92d2r" Nov 01 05:50:33.639449 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.639390 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hshw\" (UniqueName: \"kubernetes.io/projected/13895149-4860-44a1-9ecb-9d054efc3aa2-kube-api-access-9hshw\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atmqq6\" (UID: \"13895149-4860-44a1-9ecb-9d054efc3aa2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atmqq6" Nov 01 05:50:33.679962 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.678224 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5886aeb7-6e38-435a-9bd8-acb6f21ddc70-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czbwm\" (UID: \"5886aeb7-6e38-435a-9bd8-acb6f21ddc70\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czbwm" Nov 01 05:50:33.679962 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.678318 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jhwn\" (UniqueName: \"kubernetes.io/projected/5886aeb7-6e38-435a-9bd8-acb6f21ddc70-kube-api-access-2jhwn\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czbwm\" (UID: \"5886aeb7-6e38-435a-9bd8-acb6f21ddc70\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czbwm" Nov 01 05:50:33.679962 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.678398 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5886aeb7-6e38-435a-9bd8-acb6f21ddc70-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czbwm\" (UID: \"5886aeb7-6e38-435a-9bd8-acb6f21ddc70\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czbwm" Nov 01 05:50:33.679962 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.678839 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5886aeb7-6e38-435a-9bd8-acb6f21ddc70-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czbwm\" (UID: \"5886aeb7-6e38-435a-9bd8-acb6f21ddc70\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czbwm" Nov 01 05:50:33.679962 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.679109 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5886aeb7-6e38-435a-9bd8-acb6f21ddc70-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czbwm\" (UID: \"5886aeb7-6e38-435a-9bd8-acb6f21ddc70\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czbwm" Nov 01 05:50:33.709515 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.709186 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atmqq6" Nov 01 05:50:33.742923 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.742889 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jhwn\" (UniqueName: \"kubernetes.io/projected/5886aeb7-6e38-435a-9bd8-acb6f21ddc70-kube-api-access-2jhwn\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czbwm\" (UID: \"5886aeb7-6e38-435a-9bd8-acb6f21ddc70\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czbwm" Nov 01 05:50:33.811542 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.811511 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czbwm" Nov 01 05:50:33.885007 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:33.884063 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gg2m2"] Nov 01 05:50:33.888652 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:50:33.888616 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eedcd2c_e8a2_4244_a764_bb19b5aa726f.slice/crio-6e74e8bd3883e8aa37e2a02f1d1b0e16a6b7c6bf24c23783575dcb2080c12862 WatchSource:0}: Error finding container 6e74e8bd3883e8aa37e2a02f1d1b0e16a6b7c6bf24c23783575dcb2080c12862: Status 404 returned error can't find the container with id 6e74e8bd3883e8aa37e2a02f1d1b0e16a6b7c6bf24c23783575dcb2080c12862 Nov 01 05:50:34.000271 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:34.000101 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c92d2r"] Nov 01 05:50:34.003117 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:50:34.003086 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd01e10_2e4a_4d5f_abda_ea50f28586b8.slice/crio-af032002496a9be8376a66e57a45032a283971797f9ba6e6b8c727ca3177374f WatchSource:0}: Error finding container af032002496a9be8376a66e57a45032a283971797f9ba6e6b8c727ca3177374f: Status 404 returned error can't find the container with id af032002496a9be8376a66e57a45032a283971797f9ba6e6b8c727ca3177374f Nov 01 05:50:34.092022 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:34.091990 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atmqq6"] Nov 01 05:50:34.095861 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:50:34.095787 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13895149_4860_44a1_9ecb_9d054efc3aa2.slice/crio-8bda945385408852a160f74eadf91c14605d1cec5a7fbbc4f8fd35f9e8cd4bce WatchSource:0}: Error finding container 8bda945385408852a160f74eadf91c14605d1cec5a7fbbc4f8fd35f9e8cd4bce: Status 404 returned error can't find the container with id 8bda945385408852a160f74eadf91c14605d1cec5a7fbbc4f8fd35f9e8cd4bce Nov 01 05:50:34.192976 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:34.192944 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czbwm"] Nov 01 05:50:34.195770 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:50:34.195742 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5886aeb7_6e38_435a_9bd8_acb6f21ddc70.slice/crio-a2a949e7f7b9ab24ce22d3be0476eb77657723a2fd475c29deef6117ce7a9750 WatchSource:0}: Error finding container a2a949e7f7b9ab24ce22d3be0476eb77657723a2fd475c29deef6117ce7a9750: Status 404 returned error can't find the container with id a2a949e7f7b9ab24ce22d3be0476eb77657723a2fd475c29deef6117ce7a9750 Nov 01 05:50:34.745914 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:34.745880 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gdd9d"] Nov 01 05:50:34.749648 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:34.749625 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gdd9d" Nov 01 05:50:34.803387 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:34.803357 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-69k5d\"" Nov 01 05:50:34.808043 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:34.807975 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gdd9d"] Nov 01 05:50:34.887863 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:34.887821 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gg2m2" event={"ID":"1eedcd2c-e8a2-4244-a764-bb19b5aa726f","Type":"ContainerStarted","Data":"6e74e8bd3883e8aa37e2a02f1d1b0e16a6b7c6bf24c23783575dcb2080c12862"} Nov 01 05:50:34.889535 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:34.889508 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atmqq6" event={"ID":"13895149-4860-44a1-9ecb-9d054efc3aa2","Type":"ContainerStarted","Data":"8bda945385408852a160f74eadf91c14605d1cec5a7fbbc4f8fd35f9e8cd4bce"} Nov 01 05:50:34.890986 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:34.890962 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4zlj\" (UniqueName: \"kubernetes.io/projected/cb2ff142-bb4e-42b1-a697-7848669a1d61-kube-api-access-w4zlj\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gdd9d\" (UID: \"cb2ff142-bb4e-42b1-a697-7848669a1d61\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gdd9d" Nov 01 05:50:34.891109 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:34.891003 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb2ff142-bb4e-42b1-a697-7848669a1d61-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gdd9d\" (UID: \"cb2ff142-bb4e-42b1-a697-7848669a1d61\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gdd9d" Nov 01 05:50:34.891109 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:34.891066 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c92d2r" event={"ID":"4bd01e10-2e4a-4d5f-abda-ea50f28586b8","Type":"ContainerStarted","Data":"af032002496a9be8376a66e57a45032a283971797f9ba6e6b8c727ca3177374f"} Nov 01 05:50:34.891233 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:34.891121 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb2ff142-bb4e-42b1-a697-7848669a1d61-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gdd9d\" (UID: \"cb2ff142-bb4e-42b1-a697-7848669a1d61\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gdd9d" Nov 01 05:50:34.893077 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:34.893058 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czbwm" event={"ID":"5886aeb7-6e38-435a-9bd8-acb6f21ddc70","Type":"ContainerStarted","Data":"a2a949e7f7b9ab24ce22d3be0476eb77657723a2fd475c29deef6117ce7a9750"} Nov 01 05:50:34.992891 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:34.992250 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4zlj\" (UniqueName: \"kubernetes.io/projected/cb2ff142-bb4e-42b1-a697-7848669a1d61-kube-api-access-w4zlj\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gdd9d\" (UID: \"cb2ff142-bb4e-42b1-a697-7848669a1d61\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gdd9d" Nov 01 05:50:34.992891 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:34.992313 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb2ff142-bb4e-42b1-a697-7848669a1d61-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gdd9d\" (UID: \"cb2ff142-bb4e-42b1-a697-7848669a1d61\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gdd9d" Nov 01 05:50:34.992891 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:34.992364 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb2ff142-bb4e-42b1-a697-7848669a1d61-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gdd9d\" (UID: \"cb2ff142-bb4e-42b1-a697-7848669a1d61\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gdd9d" Nov 01 05:50:34.993608 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:34.993557 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb2ff142-bb4e-42b1-a697-7848669a1d61-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gdd9d\" (UID: \"cb2ff142-bb4e-42b1-a697-7848669a1d61\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gdd9d" Nov 01 05:50:34.993608 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:34.993571 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb2ff142-bb4e-42b1-a697-7848669a1d61-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gdd9d\" (UID: \"cb2ff142-bb4e-42b1-a697-7848669a1d61\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gdd9d" Nov 01 05:50:35.058506 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.057141 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4zlj\" (UniqueName: \"kubernetes.io/projected/cb2ff142-bb4e-42b1-a697-7848669a1d61-kube-api-access-w4zlj\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gdd9d\" (UID: \"cb2ff142-bb4e-42b1-a697-7848669a1d61\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gdd9d" Nov 01 05:50:35.070724 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.070487 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gdd9d" Nov 01 05:50:35.326400 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.324558 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl"] Nov 01 05:50:35.329201 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.328640 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 05:50:35.382835 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.382779 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 01 05:50:35.383019 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.382835 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-vqwlh\"" Nov 01 05:50:35.394902 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.394811 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl"] Nov 01 05:50:35.397011 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.396985 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cf4cxl"] Nov 01 05:50:35.405103 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.404682 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cf4cxl" Nov 01 05:50:35.483876 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.483837 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cf4cxl"] Nov 01 05:50:35.485230 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.485208 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-h9kfk\"" Nov 01 05:50:35.496934 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.496900 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-mgnkl\" (UID: \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 05:50:35.497076 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.496971 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6cdef65-3d1c-45ca-a939-71b40012156a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cf4cxl\" (UID: \"f6cdef65-3d1c-45ca-a939-71b40012156a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cf4cxl" Nov 01 05:50:35.497076 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.497008 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq2x2\" (UniqueName: \"kubernetes.io/projected/f6cdef65-3d1c-45ca-a939-71b40012156a-kube-api-access-qq2x2\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cf4cxl\" (UID: \"f6cdef65-3d1c-45ca-a939-71b40012156a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cf4cxl" Nov 01 05:50:35.497076 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.497034 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-mgnkl\" (UID: \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 05:50:35.497076 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.497070 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6cdef65-3d1c-45ca-a939-71b40012156a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cf4cxl\" (UID: \"f6cdef65-3d1c-45ca-a939-71b40012156a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cf4cxl" Nov 01 05:50:35.497285 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.497122 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-mgnkl\" (UID: \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 05:50:35.497285 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.497154 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-mgnkl\" (UID: \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 05:50:35.497285 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.497177 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-mgnkl\" (UID: \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 05:50:35.497285 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.497200 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-mgnkl\" (UID: \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 05:50:35.497285 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.497228 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbz7l\" (UniqueName: \"kubernetes.io/projected/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-kube-api-access-jbz7l\") pod \"maas-default-gateway-openshift-default-855f9574cf-mgnkl\" (UID: \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 05:50:35.497285 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.497274 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-mgnkl\" (UID: \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 05:50:35.497600 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.497302 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-mgnkl\" (UID: \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 05:50:35.505540 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.505515 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gdd9d"] Nov 01 05:50:35.514408 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:50:35.513785 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb2ff142_bb4e_42b1_a697_7848669a1d61.slice/crio-67022832309acfa94df2e6cd4dd74be4d2bb8a268e0f5dbedb28e915d39ccb07 WatchSource:0}: Error finding container 67022832309acfa94df2e6cd4dd74be4d2bb8a268e0f5dbedb28e915d39ccb07: Status 404 returned error can't find the container with id 67022832309acfa94df2e6cd4dd74be4d2bb8a268e0f5dbedb28e915d39ccb07 Nov 01 05:50:35.601228 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.598493 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6cdef65-3d1c-45ca-a939-71b40012156a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cf4cxl\" (UID: \"f6cdef65-3d1c-45ca-a939-71b40012156a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cf4cxl" Nov 01 05:50:35.601228 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.598552 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq2x2\" (UniqueName: \"kubernetes.io/projected/f6cdef65-3d1c-45ca-a939-71b40012156a-kube-api-access-qq2x2\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cf4cxl\" (UID: \"f6cdef65-3d1c-45ca-a939-71b40012156a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cf4cxl" Nov 01 05:50:35.601228 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.598583 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-mgnkl\" (UID: \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 05:50:35.601228 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.598610 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6cdef65-3d1c-45ca-a939-71b40012156a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cf4cxl\" (UID: \"f6cdef65-3d1c-45ca-a939-71b40012156a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cf4cxl" Nov 01 05:50:35.601228 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.598647 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-mgnkl\" (UID: \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 05:50:35.601228 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.598681 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-mgnkl\" (UID: \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 05:50:35.601228 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.598701 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-mgnkl\" (UID: \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 05:50:35.601228 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.598715 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-mgnkl\" (UID: \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 05:50:35.601228 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.598731 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbz7l\" (UniqueName: \"kubernetes.io/projected/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-kube-api-access-jbz7l\") pod \"maas-default-gateway-openshift-default-855f9574cf-mgnkl\" (UID: \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 05:50:35.601228 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.598763 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-mgnkl\" (UID: \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 05:50:35.601228 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.598784 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-mgnkl\" (UID: \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 05:50:35.601228 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.598802 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-mgnkl\" (UID: \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 05:50:35.601228 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.599531 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-mgnkl\" (UID: \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 05:50:35.601228 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.599853 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6cdef65-3d1c-45ca-a939-71b40012156a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cf4cxl\" (UID: \"f6cdef65-3d1c-45ca-a939-71b40012156a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cf4cxl" Nov 01 05:50:35.601228 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.600101 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-mgnkl\" (UID: \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 05:50:35.601228 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.600802 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-mgnkl\" (UID: \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 05:50:35.601228 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.600811 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-mgnkl\" (UID: \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 05:50:35.601228 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.601061 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6cdef65-3d1c-45ca-a939-71b40012156a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cf4cxl\" (UID: \"f6cdef65-3d1c-45ca-a939-71b40012156a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cf4cxl" Nov 01 05:50:35.602217 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.601296 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-mgnkl\" (UID: \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 05:50:35.602217 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.601408 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-mgnkl\" (UID: \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 05:50:35.610240 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.610213 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-mgnkl\" (UID: \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 05:50:35.666367 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.666331 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-mgnkl\" (UID: \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 05:50:35.667804 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.667739 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbz7l\" (UniqueName: \"kubernetes.io/projected/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-kube-api-access-jbz7l\") pod \"maas-default-gateway-openshift-default-855f9574cf-mgnkl\" (UID: \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 05:50:35.671755 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.671703 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq2x2\" (UniqueName: \"kubernetes.io/projected/f6cdef65-3d1c-45ca-a939-71b40012156a-kube-api-access-qq2x2\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cf4cxl\" (UID: \"f6cdef65-3d1c-45ca-a939-71b40012156a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cf4cxl" Nov 01 05:50:35.731242 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.730977 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cf4cxl" Nov 01 05:50:35.904811 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.904728 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gdd9d" event={"ID":"cb2ff142-bb4e-42b1-a697-7848669a1d61","Type":"ContainerStarted","Data":"67022832309acfa94df2e6cd4dd74be4d2bb8a268e0f5dbedb28e915d39ccb07"} Nov 01 05:50:35.946779 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:35.946525 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 05:50:36.126701 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:36.126254 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cf4cxl"] Nov 01 05:50:36.130389 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:50:36.130346 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6cdef65_3d1c_45ca_a939_71b40012156a.slice/crio-b341e84c10cc4737c54d0a7cf32968a821c94d1090cdb596b4def9f399476d4d WatchSource:0}: Error finding container b341e84c10cc4737c54d0a7cf32968a821c94d1090cdb596b4def9f399476d4d: Status 404 returned error can't find the container with id b341e84c10cc4737c54d0a7cf32968a821c94d1090cdb596b4def9f399476d4d Nov 01 05:50:36.355415 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:36.355387 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl"] Nov 01 05:50:36.358691 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:50:36.358661 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaa43c0f_ec86_4262_9bdc_773e9b11bdbb.slice/crio-046f4bc58fd2d7f77a6d2972df9b4d96313dc5107cf863b5572d8d27d3b2555e WatchSource:0}: Error finding container 046f4bc58fd2d7f77a6d2972df9b4d96313dc5107cf863b5572d8d27d3b2555e: Status 404 returned error can't find the container with id 046f4bc58fd2d7f77a6d2972df9b4d96313dc5107cf863b5572d8d27d3b2555e Nov 01 05:50:36.918822 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:36.918742 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cf4cxl" event={"ID":"f6cdef65-3d1c-45ca-a939-71b40012156a","Type":"ContainerStarted","Data":"b341e84c10cc4737c54d0a7cf32968a821c94d1090cdb596b4def9f399476d4d"} Nov 01 05:50:36.921273 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:36.921197 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" event={"ID":"baa43c0f-ec86-4262-9bdc-773e9b11bdbb","Type":"ContainerStarted","Data":"046f4bc58fd2d7f77a6d2972df9b4d96313dc5107cf863b5572d8d27d3b2555e"} Nov 01 05:50:40.000626 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:50:40.000582 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 05:50:42.989700 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:42.988469 5040 generic.go:356] "Generic (PLEG): container finished" podID="2e539788-0929-469e-9bc0-97cd39704834" containerID="d8916b1a85a5aec94313784258f68cd8076d5ca6e8f488cfecc2804bddd02b2a" exitCode=0 Nov 01 05:50:42.989700 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:42.988631 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644scwvg" event={"ID":"2e539788-0929-469e-9bc0-97cd39704834","Type":"ContainerDied","Data":"d8916b1a85a5aec94313784258f68cd8076d5ca6e8f488cfecc2804bddd02b2a"} Nov 01 05:50:43.967659 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:43.967615 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 05:50:43.967799 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:43.967713 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 05:50:43.967799 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:43.967753 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 05:50:43.996462 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:43.996330 5040 generic.go:356] "Generic (PLEG): container finished" podID="5886aeb7-6e38-435a-9bd8-acb6f21ddc70" containerID="746af5782dbc13cf4c13feb78c81938a5984cced06ea214c265e6709986ccfdc" exitCode=0 Nov 01 05:50:43.996462 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:43.996373 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czbwm" event={"ID":"5886aeb7-6e38-435a-9bd8-acb6f21ddc70","Type":"ContainerDied","Data":"746af5782dbc13cf4c13feb78c81938a5984cced06ea214c265e6709986ccfdc"} Nov 01 05:50:43.998642 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:43.998620 5040 generic.go:356] "Generic (PLEG): container finished" podID="1eedcd2c-e8a2-4244-a764-bb19b5aa726f" containerID="ce61f7457e9ca80dbd08fa470e56246adeb8e94f788dcd384d4fd731e92ba613" exitCode=0 Nov 01 05:50:43.999217 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:43.999198 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gg2m2" event={"ID":"1eedcd2c-e8a2-4244-a764-bb19b5aa726f","Type":"ContainerDied","Data":"ce61f7457e9ca80dbd08fa470e56246adeb8e94f788dcd384d4fd731e92ba613"} Nov 01 05:50:44.002271 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:44.002244 5040 generic.go:356] "Generic (PLEG): container finished" podID="13895149-4860-44a1-9ecb-9d054efc3aa2" containerID="85f66963e60baddec86f55172ee347889a80f4f3f790e5546af077ca240b15e2" exitCode=0 Nov 01 05:50:44.002377 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:44.002332 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atmqq6" event={"ID":"13895149-4860-44a1-9ecb-9d054efc3aa2","Type":"ContainerDied","Data":"85f66963e60baddec86f55172ee347889a80f4f3f790e5546af077ca240b15e2"} Nov 01 05:50:44.006364 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:44.005868 5040 generic.go:356] "Generic (PLEG): container finished" podID="4bd01e10-2e4a-4d5f-abda-ea50f28586b8" containerID="968b99134e4ba71af2cf615cb2bc3c3c218d52ff1c09051f2879d6bf81c863a2" exitCode=0 Nov 01 05:50:44.006364 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:44.005940 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c92d2r" event={"ID":"4bd01e10-2e4a-4d5f-abda-ea50f28586b8","Type":"ContainerDied","Data":"968b99134e4ba71af2cf615cb2bc3c3c218d52ff1c09051f2879d6bf81c863a2"} Nov 01 05:50:45.013057 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:45.012525 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" event={"ID":"baa43c0f-ec86-4262-9bdc-773e9b11bdbb","Type":"ContainerStarted","Data":"2ef15770562fae9547d3fdcef5b63227b8bef2b9218426060ac98e29e6f41618"} Nov 01 05:50:45.014419 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:45.014387 5040 generic.go:356] "Generic (PLEG): container finished" podID="cb2ff142-bb4e-42b1-a697-7848669a1d61" containerID="f862eb8b24901821261b6d36dd1ce3eb5d121fb77bd67deb3b42c995aee04d92" exitCode=0 Nov 01 05:50:45.014576 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:45.014469 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gdd9d" event={"ID":"cb2ff142-bb4e-42b1-a697-7848669a1d61","Type":"ContainerDied","Data":"f862eb8b24901821261b6d36dd1ce3eb5d121fb77bd67deb3b42c995aee04d92"} Nov 01 05:50:45.016932 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:45.016912 5040 generic.go:356] "Generic (PLEG): container finished" podID="f6cdef65-3d1c-45ca-a939-71b40012156a" containerID="a0b4708bf811b9778ec689bf838e7298eed8d84f14d0de1e969cd0cf6fe92878" exitCode=0 Nov 01 05:50:45.017010 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:45.016949 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cf4cxl" event={"ID":"f6cdef65-3d1c-45ca-a939-71b40012156a","Type":"ContainerDied","Data":"a0b4708bf811b9778ec689bf838e7298eed8d84f14d0de1e969cd0cf6fe92878"} Nov 01 05:50:45.141510 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:45.137141 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" podStartSLOduration=2.532724248 podStartE2EDuration="10.137115974s" totalImagesPullingTime="7.604391726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:50:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:50:45.134598536 +0000 UTC m=+323.789715703" watchObservedRunningTime="2026-11-01 05:50:45.137115974 +0000 UTC m=+323.792233142" Nov 01 05:50:45.947117 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:45.947089 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 05:50:45.952783 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:45.952760 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 05:50:46.024216 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:46.024111 5040 generic.go:356] "Generic (PLEG): container finished" podID="13895149-4860-44a1-9ecb-9d054efc3aa2" containerID="a2a00876bd9a58f6cdd4ca329b5fa1931640d6afa1866542e7a15a3d4ff37673" exitCode=0 Nov 01 05:50:46.024587 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:46.024207 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atmqq6" event={"ID":"13895149-4860-44a1-9ecb-9d054efc3aa2","Type":"ContainerDied","Data":"a2a00876bd9a58f6cdd4ca329b5fa1931640d6afa1866542e7a15a3d4ff37673"} Nov 01 05:50:46.029454 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:46.029430 5040 generic.go:356] "Generic (PLEG): container finished" podID="1eedcd2c-e8a2-4244-a764-bb19b5aa726f" containerID="9b6078a7ee93080ccade485dcc7fc955c97dcd5d8731c4617e04cf557cd50311" exitCode=0 Nov 01 05:50:46.030253 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:46.030228 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gg2m2" event={"ID":"1eedcd2c-e8a2-4244-a764-bb19b5aa726f","Type":"ContainerDied","Data":"9b6078a7ee93080ccade485dcc7fc955c97dcd5d8731c4617e04cf557cd50311"} Nov 01 05:50:46.030360 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:46.030291 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 05:50:46.031715 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:46.031649 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 05:50:47.040281 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:47.039285 5040 generic.go:356] "Generic (PLEG): container finished" podID="2e539788-0929-469e-9bc0-97cd39704834" containerID="5dfccaf456295b2639d814749ecea6ff03327d9e49c13236e71f3b687bbdf328" exitCode=0 Nov 01 05:50:47.040281 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:47.039427 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644scwvg" event={"ID":"2e539788-0929-469e-9bc0-97cd39704834","Type":"ContainerDied","Data":"5dfccaf456295b2639d814749ecea6ff03327d9e49c13236e71f3b687bbdf328"} Nov 01 05:50:47.042998 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:47.042970 5040 generic.go:356] "Generic (PLEG): container finished" podID="4bd01e10-2e4a-4d5f-abda-ea50f28586b8" containerID="b4cd38ee5d8fe5d8c09c94e070a777867ca39e2a869ba8b50deb7f3741f5dab7" exitCode=0 Nov 01 05:50:47.043125 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:47.043106 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c92d2r" event={"ID":"4bd01e10-2e4a-4d5f-abda-ea50f28586b8","Type":"ContainerDied","Data":"b4cd38ee5d8fe5d8c09c94e070a777867ca39e2a869ba8b50deb7f3741f5dab7"} Nov 01 05:50:47.046833 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:47.045588 5040 generic.go:356] "Generic (PLEG): container finished" podID="5886aeb7-6e38-435a-9bd8-acb6f21ddc70" containerID="7fab1b104cbd890ee48b32f8af195408dedaa6108abf5a3e5e90a91996541935" exitCode=0 Nov 01 05:50:47.046833 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:47.045660 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czbwm" event={"ID":"5886aeb7-6e38-435a-9bd8-acb6f21ddc70","Type":"ContainerDied","Data":"7fab1b104cbd890ee48b32f8af195408dedaa6108abf5a3e5e90a91996541935"} Nov 01 05:50:47.050461 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:47.050432 5040 generic.go:356] "Generic (PLEG): container finished" podID="cb2ff142-bb4e-42b1-a697-7848669a1d61" containerID="58efc591360882f668998fd3cd93d350bb3273af40c99769494cf64120664704" exitCode=0 Nov 01 05:50:47.050609 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:47.050568 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gdd9d" event={"ID":"cb2ff142-bb4e-42b1-a697-7848669a1d61","Type":"ContainerDied","Data":"58efc591360882f668998fd3cd93d350bb3273af40c99769494cf64120664704"} Nov 01 05:50:49.072410 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:49.072338 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cf4cxl" event={"ID":"f6cdef65-3d1c-45ca-a939-71b40012156a","Type":"ContainerStarted","Data":"6148f7ba0a6a420f7adec6c6df27ed6a44d087d1562f256f24c3936698bbab8b"} Nov 01 05:50:50.091289 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:50.091253 5040 generic.go:356] "Generic (PLEG): container finished" podID="f6cdef65-3d1c-45ca-a939-71b40012156a" containerID="6148f7ba0a6a420f7adec6c6df27ed6a44d087d1562f256f24c3936698bbab8b" exitCode=0 Nov 01 05:50:50.091855 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:50:50.091346 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cf4cxl" event={"ID":"f6cdef65-3d1c-45ca-a939-71b40012156a","Type":"ContainerDied","Data":"6148f7ba0a6a420f7adec6c6df27ed6a44d087d1562f256f24c3936698bbab8b"} Nov 01 05:51:01.211536 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:01.211430 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atmqq6" event={"ID":"13895149-4860-44a1-9ecb-9d054efc3aa2","Type":"ContainerStarted","Data":"272b5faec220d7e9bfc9c8f54c8e9d0a6c285913d1ced37b8ba2d5cfa4b4e840"} Nov 01 05:51:01.215038 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:01.215006 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gg2m2" event={"ID":"1eedcd2c-e8a2-4244-a764-bb19b5aa726f","Type":"ContainerStarted","Data":"2db4671cc1b7f7fd1f346512367546711c08480b9da6610db44525433ed01ec8"} Nov 01 05:51:01.456070 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:01.454894 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atmqq6" podStartSLOduration=2.5015793410000002 podStartE2EDuration="28.454876241s" totalImagesPullingTime="25.9532969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:50:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:51:01.333398616 +0000 UTC m=+339.988515784" watchObservedRunningTime="2026-11-01 05:51:01.454876241 +0000 UTC m=+340.109993410" Nov 01 05:51:01.456070 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:01.455645 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gg2m2" podStartSLOduration=2.92930464 podStartE2EDuration="28.455634398s" totalImagesPullingTime="25.526329758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:50:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:51:01.453321173 +0000 UTC m=+340.108438340" watchObservedRunningTime="2026-11-01 05:51:01.455634398 +0000 UTC m=+340.110751565" Nov 01 05:51:02.224937 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:02.224894 5040 generic.go:356] "Generic (PLEG): container finished" podID="13895149-4860-44a1-9ecb-9d054efc3aa2" containerID="272b5faec220d7e9bfc9c8f54c8e9d0a6c285913d1ced37b8ba2d5cfa4b4e840" exitCode=0 Nov 01 05:51:02.225420 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:02.224975 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atmqq6" event={"ID":"13895149-4860-44a1-9ecb-9d054efc3aa2","Type":"ContainerDied","Data":"272b5faec220d7e9bfc9c8f54c8e9d0a6c285913d1ced37b8ba2d5cfa4b4e840"} Nov 01 05:51:02.227590 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:02.227555 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c92d2r" event={"ID":"4bd01e10-2e4a-4d5f-abda-ea50f28586b8","Type":"ContainerStarted","Data":"0d033b262b994f1bcb8835c9c4aea302b201e59f77ff6a360c0030aa6040251f"} Nov 01 05:51:02.230611 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:02.230580 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gdd9d" event={"ID":"cb2ff142-bb4e-42b1-a697-7848669a1d61","Type":"ContainerStarted","Data":"21a76350554f081c6c68f3b7886347ef62e10bf647992dd2d9954dbed61e9dfb"} Nov 01 05:51:02.233178 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:02.233153 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644scwvg" event={"ID":"2e539788-0929-469e-9bc0-97cd39704834","Type":"ContainerStarted","Data":"26633e7207c53696dacb56ebec5a1619a6fdde3dfe0942b567731267b9e8e34a"} Nov 01 05:51:02.236989 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:02.236961 5040 generic.go:356] "Generic (PLEG): container finished" podID="1eedcd2c-e8a2-4244-a764-bb19b5aa726f" containerID="2db4671cc1b7f7fd1f346512367546711c08480b9da6610db44525433ed01ec8" exitCode=0 Nov 01 05:51:02.237111 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:02.237009 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gg2m2" event={"ID":"1eedcd2c-e8a2-4244-a764-bb19b5aa726f","Type":"ContainerDied","Data":"2db4671cc1b7f7fd1f346512367546711c08480b9da6610db44525433ed01ec8"} Nov 01 05:51:02.710746 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:02.710693 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c92d2r" podStartSLOduration=3.414091917 podStartE2EDuration="29.71068044s" totalImagesPullingTime="26.296588523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:50:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:51:02.58693191 +0000 UTC m=+341.242049079" watchObservedRunningTime="2026-11-01 05:51:02.71068044 +0000 UTC m=+341.365797638" Nov 01 05:51:02.711000 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:02.710975 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644scwvg" podStartSLOduration=2.937794716 podStartE2EDuration="32.710966991s" totalImagesPullingTime="29.773172275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:50:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:51:02.710256409 +0000 UTC m=+341.365373575" watchObservedRunningTime="2026-11-01 05:51:02.710966991 +0000 UTC m=+341.366084150" Nov 01 05:51:02.831143 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:02.830680 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gdd9d" podStartSLOduration=4.415321243 podStartE2EDuration="28.830665543s" totalImagesPullingTime="24.4153443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:50:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:51:02.829948769 +0000 UTC m=+341.485065936" watchObservedRunningTime="2026-11-01 05:51:02.830665543 +0000 UTC m=+341.485782734" Nov 01 05:51:03.243638 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:03.243599 5040 generic.go:356] "Generic (PLEG): container finished" podID="2e539788-0929-469e-9bc0-97cd39704834" containerID="26633e7207c53696dacb56ebec5a1619a6fdde3dfe0942b567731267b9e8e34a" exitCode=0 Nov 01 05:51:03.244058 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:03.243672 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644scwvg" event={"ID":"2e539788-0929-469e-9bc0-97cd39704834","Type":"ContainerDied","Data":"26633e7207c53696dacb56ebec5a1619a6fdde3dfe0942b567731267b9e8e34a"} Nov 01 05:51:03.245839 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:03.245815 5040 generic.go:356] "Generic (PLEG): container finished" podID="4bd01e10-2e4a-4d5f-abda-ea50f28586b8" containerID="0d033b262b994f1bcb8835c9c4aea302b201e59f77ff6a360c0030aa6040251f" exitCode=0 Nov 01 05:51:03.245985 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:03.245902 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c92d2r" event={"ID":"4bd01e10-2e4a-4d5f-abda-ea50f28586b8","Type":"ContainerDied","Data":"0d033b262b994f1bcb8835c9c4aea302b201e59f77ff6a360c0030aa6040251f"} Nov 01 05:51:03.248160 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:03.248132 5040 generic.go:356] "Generic (PLEG): container finished" podID="5886aeb7-6e38-435a-9bd8-acb6f21ddc70" containerID="9eea3a94e8c86b6700792e642b13f37b42134422291766b3aa94510b096eeaee" exitCode=0 Nov 01 05:51:03.248305 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:03.248218 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czbwm" event={"ID":"5886aeb7-6e38-435a-9bd8-acb6f21ddc70","Type":"ContainerDied","Data":"9eea3a94e8c86b6700792e642b13f37b42134422291766b3aa94510b096eeaee"} Nov 01 05:51:03.250290 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:03.250268 5040 generic.go:356] "Generic (PLEG): container finished" podID="cb2ff142-bb4e-42b1-a697-7848669a1d61" containerID="21a76350554f081c6c68f3b7886347ef62e10bf647992dd2d9954dbed61e9dfb" exitCode=0 Nov 01 05:51:03.250402 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:03.250343 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gdd9d" event={"ID":"cb2ff142-bb4e-42b1-a697-7848669a1d61","Type":"ContainerDied","Data":"21a76350554f081c6c68f3b7886347ef62e10bf647992dd2d9954dbed61e9dfb"} Nov 01 05:51:03.420360 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:03.420336 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gg2m2" Nov 01 05:51:03.432543 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:03.432522 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atmqq6" Nov 01 05:51:03.482022 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:03.482002 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13895149-4860-44a1-9ecb-9d054efc3aa2-kube-api-access-9hshw\" (UniqueName: \"kubernetes.io/projected/13895149-4860-44a1-9ecb-9d054efc3aa2-kube-api-access-9hshw\") pod \"13895149-4860-44a1-9ecb-9d054efc3aa2\" (UID: \"13895149-4860-44a1-9ecb-9d054efc3aa2\") " Nov 01 05:51:03.482101 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:03.482061 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eedcd2c-e8a2-4244-a764-bb19b5aa726f-kube-api-access-k5d2r\" (UniqueName: \"kubernetes.io/projected/1eedcd2c-e8a2-4244-a764-bb19b5aa726f-kube-api-access-k5d2r\") pod \"1eedcd2c-e8a2-4244-a764-bb19b5aa726f\" (UID: \"1eedcd2c-e8a2-4244-a764-bb19b5aa726f\") " Nov 01 05:51:03.482169 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:03.482099 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13895149-4860-44a1-9ecb-9d054efc3aa2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/13895149-4860-44a1-9ecb-9d054efc3aa2-bundle\") pod \"13895149-4860-44a1-9ecb-9d054efc3aa2\" (UID: \"13895149-4860-44a1-9ecb-9d054efc3aa2\") " Nov 01 05:51:03.482169 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:03.482153 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eedcd2c-e8a2-4244-a764-bb19b5aa726f-util\" (UniqueName: \"kubernetes.io/empty-dir/1eedcd2c-e8a2-4244-a764-bb19b5aa726f-util\") pod \"1eedcd2c-e8a2-4244-a764-bb19b5aa726f\" (UID: \"1eedcd2c-e8a2-4244-a764-bb19b5aa726f\") " Nov 01 05:51:03.482273 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:03.482180 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13895149-4860-44a1-9ecb-9d054efc3aa2-util\" (UniqueName: \"kubernetes.io/empty-dir/13895149-4860-44a1-9ecb-9d054efc3aa2-util\") pod \"13895149-4860-44a1-9ecb-9d054efc3aa2\" (UID: \"13895149-4860-44a1-9ecb-9d054efc3aa2\") " Nov 01 05:51:03.482326 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:03.482279 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eedcd2c-e8a2-4244-a764-bb19b5aa726f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1eedcd2c-e8a2-4244-a764-bb19b5aa726f-bundle\") pod \"1eedcd2c-e8a2-4244-a764-bb19b5aa726f\" (UID: \"1eedcd2c-e8a2-4244-a764-bb19b5aa726f\") " Nov 01 05:51:03.482925 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:03.482900 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eedcd2c-e8a2-4244-a764-bb19b5aa726f-bundle" pod "1eedcd2c-e8a2-4244-a764-bb19b5aa726f" (UID: "1eedcd2c-e8a2-4244-a764-bb19b5aa726f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:51:03.483060 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:03.483036 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13895149-4860-44a1-9ecb-9d054efc3aa2-bundle" pod "13895149-4860-44a1-9ecb-9d054efc3aa2" (UID: "13895149-4860-44a1-9ecb-9d054efc3aa2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:51:03.485073 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:03.485019 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13895149-4860-44a1-9ecb-9d054efc3aa2-kube-api-access-9hshw" pod "13895149-4860-44a1-9ecb-9d054efc3aa2" (UID: "13895149-4860-44a1-9ecb-9d054efc3aa2"). InnerVolumeSpecName "kube-api-access-9hshw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:51:03.485623 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:03.485601 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eedcd2c-e8a2-4244-a764-bb19b5aa726f-kube-api-access-k5d2r" pod "1eedcd2c-e8a2-4244-a764-bb19b5aa726f" (UID: "1eedcd2c-e8a2-4244-a764-bb19b5aa726f"). InnerVolumeSpecName "kube-api-access-k5d2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:51:03.490300 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:03.490269 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13895149-4860-44a1-9ecb-9d054efc3aa2-util" pod "13895149-4860-44a1-9ecb-9d054efc3aa2" (UID: "13895149-4860-44a1-9ecb-9d054efc3aa2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:51:03.490412 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:03.490392 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eedcd2c-e8a2-4244-a764-bb19b5aa726f-util" pod "1eedcd2c-e8a2-4244-a764-bb19b5aa726f" (UID: "1eedcd2c-e8a2-4244-a764-bb19b5aa726f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:51:03.583513 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:03.583465 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5d2r\" (UniqueName: \"kubernetes.io/projected/1eedcd2c-e8a2-4244-a764-bb19b5aa726f-kube-api-access-k5d2r\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:51:03.584205 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:03.584154 5040 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13895149-4860-44a1-9ecb-9d054efc3aa2-bundle\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:51:03.584205 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:03.584191 5040 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1eedcd2c-e8a2-4244-a764-bb19b5aa726f-util\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:51:03.584205 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:03.584212 5040 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13895149-4860-44a1-9ecb-9d054efc3aa2-util\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:51:03.584384 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:03.584224 5040 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1eedcd2c-e8a2-4244-a764-bb19b5aa726f-bundle\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:51:03.584384 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:03.584237 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hshw\" (UniqueName: \"kubernetes.io/projected/13895149-4860-44a1-9ecb-9d054efc3aa2-kube-api-access-9hshw\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:51:04.257442 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.257410 5040 generic.go:356] "Generic (PLEG): container finished" podID="f6cdef65-3d1c-45ca-a939-71b40012156a" containerID="29e1a76832e7ff9ce86c94359745e1a00b931eb05de753f0029dc6e68c7acc6f" exitCode=0 Nov 01 05:51:04.257890 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.257506 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cf4cxl" event={"ID":"f6cdef65-3d1c-45ca-a939-71b40012156a","Type":"ContainerDied","Data":"29e1a76832e7ff9ce86c94359745e1a00b931eb05de753f0029dc6e68c7acc6f"} Nov 01 05:51:04.259307 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.259290 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atmqq6" Nov 01 05:51:04.259409 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.259303 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atmqq6" event={"ID":"13895149-4860-44a1-9ecb-9d054efc3aa2","Type":"ContainerDied","Data":"8bda945385408852a160f74eadf91c14605d1cec5a7fbbc4f8fd35f9e8cd4bce"} Nov 01 05:51:04.259409 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.259349 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bda945385408852a160f74eadf91c14605d1cec5a7fbbc4f8fd35f9e8cd4bce" Nov 01 05:51:04.261360 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.261340 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gg2m2" Nov 01 05:51:04.261469 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.261441 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gg2m2" event={"ID":"1eedcd2c-e8a2-4244-a764-bb19b5aa726f","Type":"ContainerDied","Data":"6e74e8bd3883e8aa37e2a02f1d1b0e16a6b7c6bf24c23783575dcb2080c12862"} Nov 01 05:51:04.261544 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.261499 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e74e8bd3883e8aa37e2a02f1d1b0e16a6b7c6bf24c23783575dcb2080c12862" Nov 01 05:51:04.421399 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.421368 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c92d2r" Nov 01 05:51:04.465460 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.465429 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gdd9d" Nov 01 05:51:04.489739 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.489707 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644scwvg" Nov 01 05:51:04.500329 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.496569 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bd01e10-2e4a-4d5f-abda-ea50f28586b8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4bd01e10-2e4a-4d5f-abda-ea50f28586b8-bundle\") pod \"4bd01e10-2e4a-4d5f-abda-ea50f28586b8\" (UID: \"4bd01e10-2e4a-4d5f-abda-ea50f28586b8\") " Nov 01 05:51:04.500329 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.496671 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bd01e10-2e4a-4d5f-abda-ea50f28586b8-kube-api-access-tqx4d\" (UniqueName: \"kubernetes.io/projected/4bd01e10-2e4a-4d5f-abda-ea50f28586b8-kube-api-access-tqx4d\") pod \"4bd01e10-2e4a-4d5f-abda-ea50f28586b8\" (UID: \"4bd01e10-2e4a-4d5f-abda-ea50f28586b8\") " Nov 01 05:51:04.500329 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.496763 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bd01e10-2e4a-4d5f-abda-ea50f28586b8-util\" (UniqueName: \"kubernetes.io/empty-dir/4bd01e10-2e4a-4d5f-abda-ea50f28586b8-util\") pod \"4bd01e10-2e4a-4d5f-abda-ea50f28586b8\" (UID: \"4bd01e10-2e4a-4d5f-abda-ea50f28586b8\") " Nov 01 05:51:04.500329 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.497579 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bd01e10-2e4a-4d5f-abda-ea50f28586b8-bundle" pod "4bd01e10-2e4a-4d5f-abda-ea50f28586b8" (UID: "4bd01e10-2e4a-4d5f-abda-ea50f28586b8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:51:04.503008 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.502974 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bd01e10-2e4a-4d5f-abda-ea50f28586b8-util" pod "4bd01e10-2e4a-4d5f-abda-ea50f28586b8" (UID: "4bd01e10-2e4a-4d5f-abda-ea50f28586b8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:51:04.503827 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.503798 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bd01e10-2e4a-4d5f-abda-ea50f28586b8-kube-api-access-tqx4d" pod "4bd01e10-2e4a-4d5f-abda-ea50f28586b8" (UID: "4bd01e10-2e4a-4d5f-abda-ea50f28586b8"). InnerVolumeSpecName "kube-api-access-tqx4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:51:04.511785 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.511733 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czbwm" Nov 01 05:51:04.597595 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.597557 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb2ff142-bb4e-42b1-a697-7848669a1d61-util\" (UniqueName: \"kubernetes.io/empty-dir/cb2ff142-bb4e-42b1-a697-7848669a1d61-util\") pod \"cb2ff142-bb4e-42b1-a697-7848669a1d61\" (UID: \"cb2ff142-bb4e-42b1-a697-7848669a1d61\") " Nov 01 05:51:04.597754 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.597628 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb2ff142-bb4e-42b1-a697-7848669a1d61-bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb2ff142-bb4e-42b1-a697-7848669a1d61-bundle\") pod \"cb2ff142-bb4e-42b1-a697-7848669a1d61\" (UID: \"cb2ff142-bb4e-42b1-a697-7848669a1d61\") " Nov 01 05:51:04.597754 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.597656 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb2ff142-bb4e-42b1-a697-7848669a1d61-kube-api-access-w4zlj\" (UniqueName: \"kubernetes.io/projected/cb2ff142-bb4e-42b1-a697-7848669a1d61-kube-api-access-w4zlj\") pod \"cb2ff142-bb4e-42b1-a697-7848669a1d61\" (UID: \"cb2ff142-bb4e-42b1-a697-7848669a1d61\") " Nov 01 05:51:04.597754 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.597679 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e539788-0929-469e-9bc0-97cd39704834-kube-api-access-mhbr7\" (UniqueName: \"kubernetes.io/projected/2e539788-0929-469e-9bc0-97cd39704834-kube-api-access-mhbr7\") pod \"2e539788-0929-469e-9bc0-97cd39704834\" (UID: \"2e539788-0929-469e-9bc0-97cd39704834\") " Nov 01 05:51:04.597754 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.597700 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5886aeb7-6e38-435a-9bd8-acb6f21ddc70-kube-api-access-2jhwn\" (UniqueName: \"kubernetes.io/projected/5886aeb7-6e38-435a-9bd8-acb6f21ddc70-kube-api-access-2jhwn\") pod \"5886aeb7-6e38-435a-9bd8-acb6f21ddc70\" (UID: \"5886aeb7-6e38-435a-9bd8-acb6f21ddc70\") " Nov 01 05:51:04.597754 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.597729 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e539788-0929-469e-9bc0-97cd39704834-util\" (UniqueName: \"kubernetes.io/empty-dir/2e539788-0929-469e-9bc0-97cd39704834-util\") pod \"2e539788-0929-469e-9bc0-97cd39704834\" (UID: \"2e539788-0929-469e-9bc0-97cd39704834\") " Nov 01 05:51:04.597992 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.597814 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5886aeb7-6e38-435a-9bd8-acb6f21ddc70-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5886aeb7-6e38-435a-9bd8-acb6f21ddc70-bundle\") pod \"5886aeb7-6e38-435a-9bd8-acb6f21ddc70\" (UID: \"5886aeb7-6e38-435a-9bd8-acb6f21ddc70\") " Nov 01 05:51:04.597992 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.597896 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e539788-0929-469e-9bc0-97cd39704834-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e539788-0929-469e-9bc0-97cd39704834-bundle\") pod \"2e539788-0929-469e-9bc0-97cd39704834\" (UID: \"2e539788-0929-469e-9bc0-97cd39704834\") " Nov 01 05:51:04.597992 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.597929 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5886aeb7-6e38-435a-9bd8-acb6f21ddc70-util\" (UniqueName: \"kubernetes.io/empty-dir/5886aeb7-6e38-435a-9bd8-acb6f21ddc70-util\") pod \"5886aeb7-6e38-435a-9bd8-acb6f21ddc70\" (UID: \"5886aeb7-6e38-435a-9bd8-acb6f21ddc70\") " Nov 01 05:51:04.599220 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.598162 5040 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4bd01e10-2e4a-4d5f-abda-ea50f28586b8-bundle\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:51:04.599220 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.598188 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqx4d\" (UniqueName: \"kubernetes.io/projected/4bd01e10-2e4a-4d5f-abda-ea50f28586b8-kube-api-access-tqx4d\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:51:04.599220 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.598202 5040 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4bd01e10-2e4a-4d5f-abda-ea50f28586b8-util\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:51:04.599220 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.598633 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5886aeb7-6e38-435a-9bd8-acb6f21ddc70-bundle" pod "5886aeb7-6e38-435a-9bd8-acb6f21ddc70" (UID: "5886aeb7-6e38-435a-9bd8-acb6f21ddc70"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:51:04.599220 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.598683 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb2ff142-bb4e-42b1-a697-7848669a1d61-bundle" pod "cb2ff142-bb4e-42b1-a697-7848669a1d61" (UID: "cb2ff142-bb4e-42b1-a697-7848669a1d61"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:51:04.599220 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.599167 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e539788-0929-469e-9bc0-97cd39704834-bundle" pod "2e539788-0929-469e-9bc0-97cd39704834" (UID: "2e539788-0929-469e-9bc0-97cd39704834"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:51:04.600238 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.600203 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e539788-0929-469e-9bc0-97cd39704834-kube-api-access-mhbr7" pod "2e539788-0929-469e-9bc0-97cd39704834" (UID: "2e539788-0929-469e-9bc0-97cd39704834"). InnerVolumeSpecName "kube-api-access-mhbr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:51:04.600238 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.600204 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb2ff142-bb4e-42b1-a697-7848669a1d61-kube-api-access-w4zlj" pod "cb2ff142-bb4e-42b1-a697-7848669a1d61" (UID: "cb2ff142-bb4e-42b1-a697-7848669a1d61"). InnerVolumeSpecName "kube-api-access-w4zlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:51:04.600396 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.600317 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5886aeb7-6e38-435a-9bd8-acb6f21ddc70-kube-api-access-2jhwn" pod "5886aeb7-6e38-435a-9bd8-acb6f21ddc70" (UID: "5886aeb7-6e38-435a-9bd8-acb6f21ddc70"). InnerVolumeSpecName "kube-api-access-2jhwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:51:04.603965 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.603943 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb2ff142-bb4e-42b1-a697-7848669a1d61-util" pod "cb2ff142-bb4e-42b1-a697-7848669a1d61" (UID: "cb2ff142-bb4e-42b1-a697-7848669a1d61"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:51:04.604375 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.604353 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5886aeb7-6e38-435a-9bd8-acb6f21ddc70-util" pod "5886aeb7-6e38-435a-9bd8-acb6f21ddc70" (UID: "5886aeb7-6e38-435a-9bd8-acb6f21ddc70"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:51:04.605685 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.605665 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e539788-0929-469e-9bc0-97cd39704834-util" pod "2e539788-0929-469e-9bc0-97cd39704834" (UID: "2e539788-0929-469e-9bc0-97cd39704834"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:51:04.698903 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.698871 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jhwn\" (UniqueName: \"kubernetes.io/projected/5886aeb7-6e38-435a-9bd8-acb6f21ddc70-kube-api-access-2jhwn\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:51:04.698903 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.698901 5040 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e539788-0929-469e-9bc0-97cd39704834-util\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:51:04.699100 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.698911 5040 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5886aeb7-6e38-435a-9bd8-acb6f21ddc70-bundle\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:51:04.699100 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.698920 5040 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e539788-0929-469e-9bc0-97cd39704834-bundle\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:51:04.699100 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.698928 5040 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5886aeb7-6e38-435a-9bd8-acb6f21ddc70-util\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:51:04.699100 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.698935 5040 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb2ff142-bb4e-42b1-a697-7848669a1d61-util\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:51:04.699100 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.698943 5040 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb2ff142-bb4e-42b1-a697-7848669a1d61-bundle\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:51:04.699100 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.698951 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4zlj\" (UniqueName: \"kubernetes.io/projected/cb2ff142-bb4e-42b1-a697-7848669a1d61-kube-api-access-w4zlj\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:51:04.699100 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:04.698959 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhbr7\" (UniqueName: \"kubernetes.io/projected/2e539788-0929-469e-9bc0-97cd39704834-kube-api-access-mhbr7\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:51:05.267983 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:05.267959 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c92d2r" Nov 01 05:51:05.268494 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:05.267977 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c92d2r" event={"ID":"4bd01e10-2e4a-4d5f-abda-ea50f28586b8","Type":"ContainerDied","Data":"af032002496a9be8376a66e57a45032a283971797f9ba6e6b8c727ca3177374f"} Nov 01 05:51:05.268494 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:05.268036 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af032002496a9be8376a66e57a45032a283971797f9ba6e6b8c727ca3177374f" Nov 01 05:51:05.269879 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:05.269856 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czbwm" Nov 01 05:51:05.269992 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:05.269856 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czbwm" event={"ID":"5886aeb7-6e38-435a-9bd8-acb6f21ddc70","Type":"ContainerDied","Data":"a2a949e7f7b9ab24ce22d3be0476eb77657723a2fd475c29deef6117ce7a9750"} Nov 01 05:51:05.269992 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:05.269960 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2a949e7f7b9ab24ce22d3be0476eb77657723a2fd475c29deef6117ce7a9750" Nov 01 05:51:05.271774 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:05.271754 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gdd9d" event={"ID":"cb2ff142-bb4e-42b1-a697-7848669a1d61","Type":"ContainerDied","Data":"67022832309acfa94df2e6cd4dd74be4d2bb8a268e0f5dbedb28e915d39ccb07"} Nov 01 05:51:05.271884 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:05.271781 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gdd9d" Nov 01 05:51:05.271884 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:05.271789 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67022832309acfa94df2e6cd4dd74be4d2bb8a268e0f5dbedb28e915d39ccb07" Nov 01 05:51:05.273725 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:05.273705 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644scwvg" Nov 01 05:51:05.273816 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:05.273730 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644scwvg" event={"ID":"2e539788-0929-469e-9bc0-97cd39704834","Type":"ContainerDied","Data":"313e992b34fddf2026c78d0a3cd8a8dd52d6e65a463477f1a48eef81d3444855"} Nov 01 05:51:05.273816 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:05.273762 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="313e992b34fddf2026c78d0a3cd8a8dd52d6e65a463477f1a48eef81d3444855" Nov 01 05:51:05.408571 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:05.408544 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cf4cxl" Nov 01 05:51:05.505599 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:05.505564 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6cdef65-3d1c-45ca-a939-71b40012156a-util\" (UniqueName: \"kubernetes.io/empty-dir/f6cdef65-3d1c-45ca-a939-71b40012156a-util\") pod \"f6cdef65-3d1c-45ca-a939-71b40012156a\" (UID: \"f6cdef65-3d1c-45ca-a939-71b40012156a\") " Nov 01 05:51:05.505749 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:05.505616 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6cdef65-3d1c-45ca-a939-71b40012156a-kube-api-access-qq2x2\" (UniqueName: \"kubernetes.io/projected/f6cdef65-3d1c-45ca-a939-71b40012156a-kube-api-access-qq2x2\") pod \"f6cdef65-3d1c-45ca-a939-71b40012156a\" (UID: \"f6cdef65-3d1c-45ca-a939-71b40012156a\") " Nov 01 05:51:05.505749 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:05.505646 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6cdef65-3d1c-45ca-a939-71b40012156a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6cdef65-3d1c-45ca-a939-71b40012156a-bundle\") pod \"f6cdef65-3d1c-45ca-a939-71b40012156a\" (UID: \"f6cdef65-3d1c-45ca-a939-71b40012156a\") " Nov 01 05:51:05.506450 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:05.506417 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6cdef65-3d1c-45ca-a939-71b40012156a-bundle" pod "f6cdef65-3d1c-45ca-a939-71b40012156a" (UID: "f6cdef65-3d1c-45ca-a939-71b40012156a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:51:05.507895 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:05.507866 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6cdef65-3d1c-45ca-a939-71b40012156a-kube-api-access-qq2x2" pod "f6cdef65-3d1c-45ca-a939-71b40012156a" (UID: "f6cdef65-3d1c-45ca-a939-71b40012156a"). InnerVolumeSpecName "kube-api-access-qq2x2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:51:05.511093 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:05.511071 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6cdef65-3d1c-45ca-a939-71b40012156a-util" pod "f6cdef65-3d1c-45ca-a939-71b40012156a" (UID: "f6cdef65-3d1c-45ca-a939-71b40012156a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:51:05.606298 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:05.606262 5040 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6cdef65-3d1c-45ca-a939-71b40012156a-bundle\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:51:05.606298 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:05.606295 5040 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6cdef65-3d1c-45ca-a939-71b40012156a-util\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:51:05.606298 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:05.606305 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq2x2\" (UniqueName: \"kubernetes.io/projected/f6cdef65-3d1c-45ca-a939-71b40012156a-kube-api-access-qq2x2\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:51:06.280047 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:06.280018 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cf4cxl" Nov 01 05:51:06.280047 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:06.280028 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cf4cxl" event={"ID":"f6cdef65-3d1c-45ca-a939-71b40012156a","Type":"ContainerDied","Data":"b341e84c10cc4737c54d0a7cf32968a821c94d1090cdb596b4def9f399476d4d"} Nov 01 05:51:06.280677 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:06.280071 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b341e84c10cc4737c54d0a7cf32968a821c94d1090cdb596b4def9f399476d4d" Nov 01 05:51:07.887017 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.886983 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-5t7lh"] Nov 01 05:51:07.887400 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887283 5040 cpu_manager.go:414] "containerMap: removing container" podUID="f6cdef65-3d1c-45ca-a939-71b40012156a" containerName="util" Nov 01 05:51:07.887400 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887294 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6cdef65-3d1c-45ca-a939-71b40012156a" containerName="util" Nov 01 05:51:07.887400 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887304 5040 cpu_manager.go:414] "containerMap: removing container" podUID="5886aeb7-6e38-435a-9bd8-acb6f21ddc70" containerName="extract" Nov 01 05:51:07.887400 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887310 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5886aeb7-6e38-435a-9bd8-acb6f21ddc70" containerName="extract" Nov 01 05:51:07.887400 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887317 5040 cpu_manager.go:414] "containerMap: removing container" podUID="f6cdef65-3d1c-45ca-a939-71b40012156a" containerName="extract" Nov 01 05:51:07.887400 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887321 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6cdef65-3d1c-45ca-a939-71b40012156a" containerName="extract" Nov 01 05:51:07.887400 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887329 5040 cpu_manager.go:414] "containerMap: removing container" podUID="5886aeb7-6e38-435a-9bd8-acb6f21ddc70" containerName="util" Nov 01 05:51:07.887400 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887334 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5886aeb7-6e38-435a-9bd8-acb6f21ddc70" containerName="util" Nov 01 05:51:07.887400 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887341 5040 cpu_manager.go:414] "containerMap: removing container" podUID="f6cdef65-3d1c-45ca-a939-71b40012156a" containerName="pull" Nov 01 05:51:07.887400 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887346 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6cdef65-3d1c-45ca-a939-71b40012156a" containerName="pull" Nov 01 05:51:07.887400 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887352 5040 cpu_manager.go:414] "containerMap: removing container" podUID="4bd01e10-2e4a-4d5f-abda-ea50f28586b8" containerName="extract" Nov 01 05:51:07.887400 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887356 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bd01e10-2e4a-4d5f-abda-ea50f28586b8" containerName="extract" Nov 01 05:51:07.887400 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887366 5040 cpu_manager.go:414] "containerMap: removing container" podUID="4bd01e10-2e4a-4d5f-abda-ea50f28586b8" containerName="pull" Nov 01 05:51:07.887400 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887371 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bd01e10-2e4a-4d5f-abda-ea50f28586b8" containerName="pull" Nov 01 05:51:07.887400 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887377 5040 cpu_manager.go:414] "containerMap: removing container" podUID="cb2ff142-bb4e-42b1-a697-7848669a1d61" containerName="extract" Nov 01 05:51:07.887400 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887381 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb2ff142-bb4e-42b1-a697-7848669a1d61" containerName="extract" Nov 01 05:51:07.887400 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887390 5040 cpu_manager.go:414] "containerMap: removing container" podUID="2e539788-0929-469e-9bc0-97cd39704834" containerName="util" Nov 01 05:51:07.887400 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887394 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e539788-0929-469e-9bc0-97cd39704834" containerName="util" Nov 01 05:51:07.887400 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887403 5040 cpu_manager.go:414] "containerMap: removing container" podUID="13895149-4860-44a1-9ecb-9d054efc3aa2" containerName="extract" Nov 01 05:51:07.887400 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887408 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13895149-4860-44a1-9ecb-9d054efc3aa2" containerName="extract" Nov 01 05:51:07.888014 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887414 5040 cpu_manager.go:414] "containerMap: removing container" podUID="2e539788-0929-469e-9bc0-97cd39704834" containerName="extract" Nov 01 05:51:07.888014 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887418 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e539788-0929-469e-9bc0-97cd39704834" containerName="extract" Nov 01 05:51:07.888014 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887427 5040 cpu_manager.go:414] "containerMap: removing container" podUID="1eedcd2c-e8a2-4244-a764-bb19b5aa726f" containerName="util" Nov 01 05:51:07.888014 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887431 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eedcd2c-e8a2-4244-a764-bb19b5aa726f" containerName="util" Nov 01 05:51:07.888014 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887438 5040 cpu_manager.go:414] "containerMap: removing container" podUID="1eedcd2c-e8a2-4244-a764-bb19b5aa726f" containerName="pull" Nov 01 05:51:07.888014 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887442 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eedcd2c-e8a2-4244-a764-bb19b5aa726f" containerName="pull" Nov 01 05:51:07.888014 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887455 5040 cpu_manager.go:414] "containerMap: removing container" podUID="cb2ff142-bb4e-42b1-a697-7848669a1d61" containerName="util" Nov 01 05:51:07.888014 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887459 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb2ff142-bb4e-42b1-a697-7848669a1d61" containerName="util" Nov 01 05:51:07.888014 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887464 5040 cpu_manager.go:414] "containerMap: removing container" podUID="13895149-4860-44a1-9ecb-9d054efc3aa2" containerName="pull" Nov 01 05:51:07.888014 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887468 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13895149-4860-44a1-9ecb-9d054efc3aa2" containerName="pull" Nov 01 05:51:07.888014 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887474 5040 cpu_manager.go:414] "containerMap: removing container" podUID="4bd01e10-2e4a-4d5f-abda-ea50f28586b8" containerName="util" Nov 01 05:51:07.888014 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887491 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bd01e10-2e4a-4d5f-abda-ea50f28586b8" containerName="util" Nov 01 05:51:07.888014 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887496 5040 cpu_manager.go:414] "containerMap: removing container" podUID="13895149-4860-44a1-9ecb-9d054efc3aa2" containerName="util" Nov 01 05:51:07.888014 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887500 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13895149-4860-44a1-9ecb-9d054efc3aa2" containerName="util" Nov 01 05:51:07.888014 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887507 5040 cpu_manager.go:414] "containerMap: removing container" podUID="cb2ff142-bb4e-42b1-a697-7848669a1d61" containerName="pull" Nov 01 05:51:07.888014 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887511 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb2ff142-bb4e-42b1-a697-7848669a1d61" containerName="pull" Nov 01 05:51:07.888014 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887516 5040 cpu_manager.go:414] "containerMap: removing container" podUID="1eedcd2c-e8a2-4244-a764-bb19b5aa726f" containerName="extract" Nov 01 05:51:07.888014 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887520 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eedcd2c-e8a2-4244-a764-bb19b5aa726f" containerName="extract" Nov 01 05:51:07.888014 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887527 5040 cpu_manager.go:414] "containerMap: removing container" podUID="5886aeb7-6e38-435a-9bd8-acb6f21ddc70" containerName="pull" Nov 01 05:51:07.888014 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887532 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5886aeb7-6e38-435a-9bd8-acb6f21ddc70" containerName="pull" Nov 01 05:51:07.888014 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887538 5040 cpu_manager.go:414] "containerMap: removing container" podUID="2e539788-0929-469e-9bc0-97cd39704834" containerName="pull" Nov 01 05:51:07.888014 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887542 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e539788-0929-469e-9bc0-97cd39704834" containerName="pull" Nov 01 05:51:07.888014 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887588 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb2ff142-bb4e-42b1-a697-7848669a1d61" containerName="extract" Nov 01 05:51:07.888014 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887597 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="13895149-4860-44a1-9ecb-9d054efc3aa2" containerName="extract" Nov 01 05:51:07.888014 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887602 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e539788-0929-469e-9bc0-97cd39704834" containerName="extract" Nov 01 05:51:07.888014 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887610 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eedcd2c-e8a2-4244-a764-bb19b5aa726f" containerName="extract" Nov 01 05:51:07.888014 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887616 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="5886aeb7-6e38-435a-9bd8-acb6f21ddc70" containerName="extract" Nov 01 05:51:07.888014 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887622 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6cdef65-3d1c-45ca-a939-71b40012156a" containerName="extract" Nov 01 05:51:07.888014 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.887628 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bd01e10-2e4a-4d5f-abda-ea50f28586b8" containerName="extract" Nov 01 05:51:07.889590 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.889574 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-5t7lh" Nov 01 05:51:07.946677 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.944120 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 01 05:51:07.946677 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.944217 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 01 05:51:07.946677 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.944314 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 01 05:51:07.946677 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.944407 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 01 05:51:07.994470 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.994435 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-cq826\"" Nov 01 05:51:07.994470 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:07.994436 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 01 05:51:08.011801 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:08.011772 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-5t7lh"] Nov 01 05:51:08.026245 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:08.026219 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zng8\" (UniqueName: \"kubernetes.io/projected/271701b7-e770-4e41-ae34-f23b153c5061-kube-api-access-4zng8\") pod \"opendatahub-operator-controller-manager-985c476f9-5t7lh\" (UID: \"271701b7-e770-4e41-ae34-f23b153c5061\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-5t7lh" Nov 01 05:51:08.026392 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:08.026254 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/271701b7-e770-4e41-ae34-f23b153c5061-tmp\") pod \"opendatahub-operator-controller-manager-985c476f9-5t7lh\" (UID: \"271701b7-e770-4e41-ae34-f23b153c5061\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-5t7lh" Nov 01 05:51:08.026392 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:08.026341 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/271701b7-e770-4e41-ae34-f23b153c5061-manifests\") pod \"opendatahub-operator-controller-manager-985c476f9-5t7lh\" (UID: \"271701b7-e770-4e41-ae34-f23b153c5061\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-5t7lh" Nov 01 05:51:08.026392 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:08.026374 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/271701b7-e770-4e41-ae34-f23b153c5061-webhook-cert\") pod \"opendatahub-operator-controller-manager-985c476f9-5t7lh\" (UID: \"271701b7-e770-4e41-ae34-f23b153c5061\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-5t7lh" Nov 01 05:51:08.026516 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:08.026396 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/271701b7-e770-4e41-ae34-f23b153c5061-metrics-cert\") pod \"opendatahub-operator-controller-manager-985c476f9-5t7lh\" (UID: \"271701b7-e770-4e41-ae34-f23b153c5061\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-5t7lh" Nov 01 05:51:08.026516 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:08.026444 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/271701b7-e770-4e41-ae34-f23b153c5061-apiservice-cert\") pod \"opendatahub-operator-controller-manager-985c476f9-5t7lh\" (UID: \"271701b7-e770-4e41-ae34-f23b153c5061\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-5t7lh" Nov 01 05:51:08.126947 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:08.126914 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/271701b7-e770-4e41-ae34-f23b153c5061-metrics-cert\") pod \"opendatahub-operator-controller-manager-985c476f9-5t7lh\" (UID: \"271701b7-e770-4e41-ae34-f23b153c5061\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-5t7lh" Nov 01 05:51:08.127121 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:08.126950 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/271701b7-e770-4e41-ae34-f23b153c5061-apiservice-cert\") pod \"opendatahub-operator-controller-manager-985c476f9-5t7lh\" (UID: \"271701b7-e770-4e41-ae34-f23b153c5061\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-5t7lh" Nov 01 05:51:08.127121 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:08.127101 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zng8\" (UniqueName: \"kubernetes.io/projected/271701b7-e770-4e41-ae34-f23b153c5061-kube-api-access-4zng8\") pod \"opendatahub-operator-controller-manager-985c476f9-5t7lh\" (UID: \"271701b7-e770-4e41-ae34-f23b153c5061\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-5t7lh" Nov 01 05:51:08.127193 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:08.127145 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/271701b7-e770-4e41-ae34-f23b153c5061-tmp\") pod \"opendatahub-operator-controller-manager-985c476f9-5t7lh\" (UID: \"271701b7-e770-4e41-ae34-f23b153c5061\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-5t7lh" Nov 01 05:51:08.127231 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:08.127201 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/271701b7-e770-4e41-ae34-f23b153c5061-manifests\") pod \"opendatahub-operator-controller-manager-985c476f9-5t7lh\" (UID: \"271701b7-e770-4e41-ae34-f23b153c5061\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-5t7lh" Nov 01 05:51:08.127264 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:08.127231 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/271701b7-e770-4e41-ae34-f23b153c5061-webhook-cert\") pod \"opendatahub-operator-controller-manager-985c476f9-5t7lh\" (UID: \"271701b7-e770-4e41-ae34-f23b153c5061\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-5t7lh" Nov 01 05:51:08.127707 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:08.127680 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/271701b7-e770-4e41-ae34-f23b153c5061-manifests\") pod \"opendatahub-operator-controller-manager-985c476f9-5t7lh\" (UID: \"271701b7-e770-4e41-ae34-f23b153c5061\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-5t7lh" Nov 01 05:51:08.129532 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:08.129503 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/271701b7-e770-4e41-ae34-f23b153c5061-tmp\") pod \"opendatahub-operator-controller-manager-985c476f9-5t7lh\" (UID: \"271701b7-e770-4e41-ae34-f23b153c5061\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-5t7lh" Nov 01 05:51:08.129643 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:08.129550 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/271701b7-e770-4e41-ae34-f23b153c5061-apiservice-cert\") pod \"opendatahub-operator-controller-manager-985c476f9-5t7lh\" (UID: \"271701b7-e770-4e41-ae34-f23b153c5061\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-5t7lh" Nov 01 05:51:08.129710 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:08.129672 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/271701b7-e770-4e41-ae34-f23b153c5061-webhook-cert\") pod \"opendatahub-operator-controller-manager-985c476f9-5t7lh\" (UID: \"271701b7-e770-4e41-ae34-f23b153c5061\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-5t7lh" Nov 01 05:51:08.129710 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:08.129681 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/271701b7-e770-4e41-ae34-f23b153c5061-metrics-cert\") pod \"opendatahub-operator-controller-manager-985c476f9-5t7lh\" (UID: \"271701b7-e770-4e41-ae34-f23b153c5061\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-5t7lh" Nov 01 05:51:08.235866 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:08.235788 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zng8\" (UniqueName: \"kubernetes.io/projected/271701b7-e770-4e41-ae34-f23b153c5061-kube-api-access-4zng8\") pod \"opendatahub-operator-controller-manager-985c476f9-5t7lh\" (UID: \"271701b7-e770-4e41-ae34-f23b153c5061\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-5t7lh" Nov 01 05:51:08.501304 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:08.501267 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-5t7lh" Nov 01 05:51:09.040353 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:51:09.040319 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod271701b7_e770_4e41_ae34_f23b153c5061.slice/crio-f9f3c247b9c10eb1129c62bd7382fbec21159796add86b0a1ef227d7b3e381e3 WatchSource:0}: Error finding container f9f3c247b9c10eb1129c62bd7382fbec21159796add86b0a1ef227d7b3e381e3: Status 404 returned error can't find the container with id f9f3c247b9c10eb1129c62bd7382fbec21159796add86b0a1ef227d7b3e381e3 Nov 01 05:51:09.088836 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:09.088797 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-5t7lh"] Nov 01 05:51:09.294065 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:09.293971 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-5t7lh" event={"ID":"271701b7-e770-4e41-ae34-f23b153c5061","Type":"ContainerStarted","Data":"f9f3c247b9c10eb1129c62bd7382fbec21159796add86b0a1ef227d7b3e381e3"} Nov 01 05:51:09.975768 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:51:09.975734 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 05:51:13.314414 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:13.314377 5040 generic.go:356] "Generic (PLEG): container finished" podID="271701b7-e770-4e41-ae34-f23b153c5061" containerID="47c11e8e7b5216c2f05627eb20cd87a11cde22a0afb8e5c3457ee6cb2e89bdca" exitCode=0 Nov 01 05:51:13.314872 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:13.314475 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-5t7lh" event={"ID":"271701b7-e770-4e41-ae34-f23b153c5061","Type":"ContainerDied","Data":"47c11e8e7b5216c2f05627eb20cd87a11cde22a0afb8e5c3457ee6cb2e89bdca"} Nov 01 05:51:14.321227 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:14.321186 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-5t7lh" event={"ID":"271701b7-e770-4e41-ae34-f23b153c5061","Type":"ContainerStarted","Data":"bf59dde5a107dbb6a1be0081ff702dba9e6c754fbdb02841a16457e67cc10acb"} Nov 01 05:51:14.321735 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:14.321258 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-5t7lh" Nov 01 05:51:14.562964 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:14.562909 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-5t7lh" podStartSLOduration=3.077825108 podStartE2EDuration="7.562892897s" totalImagesPullingTime="4.03333224s" totalInitContainerRuntime="451.735549ms" isStatefulPod=false podCreationTimestamp="2026-11-01 05:51:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:51:14.511196417 +0000 UTC m=+353.166313608" watchObservedRunningTime="2026-11-01 05:51:14.562892897 +0000 UTC m=+353.218010064" Nov 01 05:51:15.875945 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:15.875908 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-p8pjd"] Nov 01 05:51:15.878433 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:15.878412 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-p8pjd" Nov 01 05:51:15.939643 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:15.939603 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-p8pjd"] Nov 01 05:51:15.987667 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:15.987619 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 01 05:51:15.987667 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:15.987620 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 01 05:51:15.987873 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:15.987673 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-4txnk\"" Nov 01 05:51:15.992005 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:15.991985 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcpgp\" (UniqueName: \"kubernetes.io/projected/58847028-6023-4f04-bd0d-9d6a4327b694-kube-api-access-zcpgp\") pod \"limitador-operator-controller-manager-5778bb4449-p8pjd\" (UID: \"58847028-6023-4f04-bd0d-9d6a4327b694\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-p8pjd" Nov 01 05:51:16.092869 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:16.092839 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcpgp\" (UniqueName: \"kubernetes.io/projected/58847028-6023-4f04-bd0d-9d6a4327b694-kube-api-access-zcpgp\") pod \"limitador-operator-controller-manager-5778bb4449-p8pjd\" (UID: \"58847028-6023-4f04-bd0d-9d6a4327b694\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-p8pjd" Nov 01 05:51:16.154525 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:16.154441 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcpgp\" (UniqueName: \"kubernetes.io/projected/58847028-6023-4f04-bd0d-9d6a4327b694-kube-api-access-zcpgp\") pod \"limitador-operator-controller-manager-5778bb4449-p8pjd\" (UID: \"58847028-6023-4f04-bd0d-9d6a4327b694\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-p8pjd" Nov 01 05:51:16.189314 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:16.189279 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-p8pjd" Nov 01 05:51:16.562163 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:16.562133 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-p8pjd"] Nov 01 05:51:16.563672 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:51:16.563647 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58847028_6023_4f04_bd0d_9d6a4327b694.slice/crio-8898947144bcfaaed4e13ca641eeba3456307822687e7e3ba4fdfd5af1735621 WatchSource:0}: Error finding container 8898947144bcfaaed4e13ca641eeba3456307822687e7e3ba4fdfd5af1735621: Status 404 returned error can't find the container with id 8898947144bcfaaed4e13ca641eeba3456307822687e7e3ba4fdfd5af1735621 Nov 01 05:51:17.335319 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:17.335283 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-p8pjd" event={"ID":"58847028-6023-4f04-bd0d-9d6a4327b694","Type":"ContainerStarted","Data":"8898947144bcfaaed4e13ca641eeba3456307822687e7e3ba4fdfd5af1735621"} Nov 01 05:51:20.352305 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:20.352214 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-p8pjd" event={"ID":"58847028-6023-4f04-bd0d-9d6a4327b694","Type":"ContainerStarted","Data":"efbf87dcd031d5e97f95266571d94e001063199c788d061f0ce9814778b05c24"} Nov 01 05:51:20.352713 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:20.352411 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-p8pjd" Nov 01 05:51:20.469621 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:20.469568 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-p8pjd" podStartSLOduration=2.001355072 podStartE2EDuration="5.469553425s" totalImagesPullingTime="3.468198353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:51:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:51:20.469360337 +0000 UTC m=+359.124477506" watchObservedRunningTime="2026-11-01 05:51:20.469553425 +0000 UTC m=+359.124670591" Nov 01 05:51:21.421754 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:21.421722 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-p4xhq"] Nov 01 05:51:21.423893 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:21.423878 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-p4xhq" Nov 01 05:51:21.477098 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:21.477060 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-wm74n\"" Nov 01 05:51:21.529331 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:21.529298 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-p4xhq"] Nov 01 05:51:21.540895 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:21.540871 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbnwd\" (UniqueName: \"kubernetes.io/projected/4e64dd96-b5ba-4115-b87d-ffa91d637530-kube-api-access-kbnwd\") pod \"authorino-operator-7b7bb4bc77-p4xhq\" (UID: \"4e64dd96-b5ba-4115-b87d-ffa91d637530\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-p4xhq" Nov 01 05:51:21.642302 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:21.642259 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbnwd\" (UniqueName: \"kubernetes.io/projected/4e64dd96-b5ba-4115-b87d-ffa91d637530-kube-api-access-kbnwd\") pod \"authorino-operator-7b7bb4bc77-p4xhq\" (UID: \"4e64dd96-b5ba-4115-b87d-ffa91d637530\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-p4xhq" Nov 01 05:51:21.704060 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:21.703989 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbnwd\" (UniqueName: \"kubernetes.io/projected/4e64dd96-b5ba-4115-b87d-ffa91d637530-kube-api-access-kbnwd\") pod \"authorino-operator-7b7bb4bc77-p4xhq\" (UID: \"4e64dd96-b5ba-4115-b87d-ffa91d637530\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-p4xhq" Nov 01 05:51:21.735017 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:21.734986 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-p4xhq" Nov 01 05:51:22.110219 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:22.110194 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-p4xhq"] Nov 01 05:51:22.112227 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:51:22.112178 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e64dd96_b5ba_4115_b87d_ffa91d637530.slice/crio-2344fc2061619156e1ebe2327fd65660f7367771680ed4b908ed02b264519293 WatchSource:0}: Error finding container 2344fc2061619156e1ebe2327fd65660f7367771680ed4b908ed02b264519293: Status 404 returned error can't find the container with id 2344fc2061619156e1ebe2327fd65660f7367771680ed4b908ed02b264519293 Nov 01 05:51:22.363108 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:22.363025 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-p4xhq" event={"ID":"4e64dd96-b5ba-4115-b87d-ffa91d637530","Type":"ContainerStarted","Data":"2344fc2061619156e1ebe2327fd65660f7367771680ed4b908ed02b264519293"} Nov 01 05:51:25.328511 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:25.328455 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-5t7lh" Nov 01 05:51:25.377716 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:25.377681 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-p4xhq" event={"ID":"4e64dd96-b5ba-4115-b87d-ffa91d637530","Type":"ContainerStarted","Data":"c31e040f008e3b97312d9aa7a849b35ad79d805d01e7ba50d64a495d7cb5e69e"} Nov 01 05:51:25.377911 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:25.377827 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-p4xhq" Nov 01 05:51:25.640284 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:25.640174 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-p4xhq" podStartSLOduration=1.7654721690000001 podStartE2EDuration="4.640160312s" totalImagesPullingTime="2.874688143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:51:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:51:25.638393338 +0000 UTC m=+364.293510507" watchObservedRunningTime="2026-11-01 05:51:25.640160312 +0000 UTC m=+364.295277478" Nov 01 05:51:25.744526 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:25.744472 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-6c4qz"] Nov 01 05:51:25.748930 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:25.748908 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6c4qz" Nov 01 05:51:25.807273 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:25.807246 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-6c4qz"] Nov 01 05:51:25.808116 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:25.808101 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 01 05:51:25.808222 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:25.808175 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-klh5r\"" Nov 01 05:51:25.879381 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:25.879350 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzc5b\" (UniqueName: \"kubernetes.io/projected/da0c8971-758e-4a48-9f10-4c90ed9b4f1a-kube-api-access-xzc5b\") pod \"dns-operator-controller-manager-8557cbdb84-6c4qz\" (UID: \"da0c8971-758e-4a48-9f10-4c90ed9b4f1a\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6c4qz" Nov 01 05:51:25.980907 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:25.980816 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzc5b\" (UniqueName: \"kubernetes.io/projected/da0c8971-758e-4a48-9f10-4c90ed9b4f1a-kube-api-access-xzc5b\") pod \"dns-operator-controller-manager-8557cbdb84-6c4qz\" (UID: \"da0c8971-758e-4a48-9f10-4c90ed9b4f1a\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6c4qz" Nov 01 05:51:26.044640 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:26.044602 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzc5b\" (UniqueName: \"kubernetes.io/projected/da0c8971-758e-4a48-9f10-4c90ed9b4f1a-kube-api-access-xzc5b\") pod \"dns-operator-controller-manager-8557cbdb84-6c4qz\" (UID: \"da0c8971-758e-4a48-9f10-4c90ed9b4f1a\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6c4qz" Nov 01 05:51:26.061977 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:26.061953 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6c4qz" Nov 01 05:51:26.434239 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:26.434214 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-6c4qz"] Nov 01 05:51:26.436503 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:51:26.436464 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda0c8971_758e_4a48_9f10_4c90ed9b4f1a.slice/crio-fc8c7073cfcbed2e897c97b4fcdee2b471d2d1f6c5e9aba533e40e3228b81e66 WatchSource:0}: Error finding container fc8c7073cfcbed2e897c97b4fcdee2b471d2d1f6c5e9aba533e40e3228b81e66: Status 404 returned error can't find the container with id fc8c7073cfcbed2e897c97b4fcdee2b471d2d1f6c5e9aba533e40e3228b81e66 Nov 01 05:51:27.389552 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:27.389516 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6c4qz" event={"ID":"da0c8971-758e-4a48-9f10-4c90ed9b4f1a","Type":"ContainerStarted","Data":"fc8c7073cfcbed2e897c97b4fcdee2b471d2d1f6c5e9aba533e40e3228b81e66"} Nov 01 05:51:29.402207 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:29.402171 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6c4qz" event={"ID":"da0c8971-758e-4a48-9f10-4c90ed9b4f1a","Type":"ContainerStarted","Data":"751f7105192ae3e9a6131cc458c3890f787923a832bac8dadda8d3e2052ce034"} Nov 01 05:51:29.402674 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:29.402334 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6c4qz" Nov 01 05:51:29.459174 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:29.459086 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dmsrp"] Nov 01 05:51:29.462643 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:29.462622 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dmsrp" Nov 01 05:51:29.516136 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:29.516107 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 01 05:51:29.516345 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:29.516213 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 01 05:51:29.516345 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:29.516260 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-9tkcq\"" Nov 01 05:51:29.518140 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:29.518112 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dmsrp"] Nov 01 05:51:29.568609 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:29.568560 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6c4qz" podStartSLOduration=1.8518467379999999 podStartE2EDuration="4.568545531s" totalImagesPullingTime="2.716698793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:51:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:51:29.566811014 +0000 UTC m=+368.221928181" watchObservedRunningTime="2026-11-01 05:51:29.568545531 +0000 UTC m=+368.223662698" Nov 01 05:51:29.610128 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:29.610082 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aed91991-16d6-4b3a-9e79-7c1f6106af4a-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dmsrp\" (UID: \"aed91991-16d6-4b3a-9e79-7c1f6106af4a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dmsrp" Nov 01 05:51:29.610396 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:29.610188 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf5lg\" (UniqueName: \"kubernetes.io/projected/aed91991-16d6-4b3a-9e79-7c1f6106af4a-kube-api-access-vf5lg\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dmsrp\" (UID: \"aed91991-16d6-4b3a-9e79-7c1f6106af4a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dmsrp" Nov 01 05:51:29.711454 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:29.711363 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aed91991-16d6-4b3a-9e79-7c1f6106af4a-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dmsrp\" (UID: \"aed91991-16d6-4b3a-9e79-7c1f6106af4a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dmsrp" Nov 01 05:51:29.711454 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:29.711405 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf5lg\" (UniqueName: \"kubernetes.io/projected/aed91991-16d6-4b3a-9e79-7c1f6106af4a-kube-api-access-vf5lg\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dmsrp\" (UID: \"aed91991-16d6-4b3a-9e79-7c1f6106af4a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dmsrp" Nov 01 05:51:29.711831 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:29.711810 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aed91991-16d6-4b3a-9e79-7c1f6106af4a-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dmsrp\" (UID: \"aed91991-16d6-4b3a-9e79-7c1f6106af4a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dmsrp" Nov 01 05:51:29.773237 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:29.773205 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf5lg\" (UniqueName: \"kubernetes.io/projected/aed91991-16d6-4b3a-9e79-7c1f6106af4a-kube-api-access-vf5lg\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dmsrp\" (UID: \"aed91991-16d6-4b3a-9e79-7c1f6106af4a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dmsrp" Nov 01 05:51:29.773958 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:29.773942 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dmsrp" Nov 01 05:51:30.199025 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:30.198998 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dmsrp"] Nov 01 05:51:30.200727 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:51:30.200693 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaed91991_16d6_4b3a_9e79_7c1f6106af4a.slice/crio-7a766cbcdf68fb4a65541bcbda0291a25cb34cb459d1b4514d252f3d2b709113 WatchSource:0}: Error finding container 7a766cbcdf68fb4a65541bcbda0291a25cb34cb459d1b4514d252f3d2b709113: Status 404 returned error can't find the container with id 7a766cbcdf68fb4a65541bcbda0291a25cb34cb459d1b4514d252f3d2b709113 Nov 01 05:51:30.408012 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:30.407974 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dmsrp" event={"ID":"aed91991-16d6-4b3a-9e79-7c1f6106af4a","Type":"ContainerStarted","Data":"7a766cbcdf68fb4a65541bcbda0291a25cb34cb459d1b4514d252f3d2b709113"} Nov 01 05:51:31.359711 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:31.359677 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-p8pjd" Nov 01 05:51:34.431087 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:34.431052 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dmsrp" event={"ID":"aed91991-16d6-4b3a-9e79-7c1f6106af4a","Type":"ContainerStarted","Data":"41d24c15f493bd556d50dbd4f39a01d16368989dc893451b86042f7ba39b4232"} Nov 01 05:51:34.653648 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:34.653586 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dmsrp" podStartSLOduration=2.471806483 podStartE2EDuration="5.653569378s" totalImagesPullingTime="3.181762895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:51:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:51:34.602753978 +0000 UTC m=+373.257871146" watchObservedRunningTime="2026-11-01 05:51:34.653569378 +0000 UTC m=+373.308686544" Nov 01 05:51:36.084757 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:36.084722 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-brtbn"] Nov 01 05:51:36.087127 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:36.087110 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-brtbn" Nov 01 05:51:36.143740 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:36.143717 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 01 05:51:36.194606 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:36.194567 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 01 05:51:36.194759 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:36.194614 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-dqpdv\"" Nov 01 05:51:36.196810 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:36.196787 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-brtbn"] Nov 01 05:51:36.266550 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:36.266523 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f94f4767-3f84-429a-bff5-38c4fa82deaa-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-brtbn\" (UID: \"f94f4767-3f84-429a-bff5-38c4fa82deaa\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-brtbn" Nov 01 05:51:36.266669 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:36.266566 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6hg5\" (UniqueName: \"kubernetes.io/projected/f94f4767-3f84-429a-bff5-38c4fa82deaa-kube-api-access-q6hg5\") pod \"cert-manager-webhook-5dbbbf75cc-brtbn\" (UID: \"f94f4767-3f84-429a-bff5-38c4fa82deaa\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-brtbn" Nov 01 05:51:36.367070 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:36.366996 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6hg5\" (UniqueName: \"kubernetes.io/projected/f94f4767-3f84-429a-bff5-38c4fa82deaa-kube-api-access-q6hg5\") pod \"cert-manager-webhook-5dbbbf75cc-brtbn\" (UID: \"f94f4767-3f84-429a-bff5-38c4fa82deaa\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-brtbn" Nov 01 05:51:36.367212 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:36.367074 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f94f4767-3f84-429a-bff5-38c4fa82deaa-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-brtbn\" (UID: \"f94f4767-3f84-429a-bff5-38c4fa82deaa\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-brtbn" Nov 01 05:51:36.385665 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:36.385638 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-p4xhq" Nov 01 05:51:36.428959 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:36.428926 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f94f4767-3f84-429a-bff5-38c4fa82deaa-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-brtbn\" (UID: \"f94f4767-3f84-429a-bff5-38c4fa82deaa\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-brtbn" Nov 01 05:51:36.429091 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:36.428985 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6hg5\" (UniqueName: \"kubernetes.io/projected/f94f4767-3f84-429a-bff5-38c4fa82deaa-kube-api-access-q6hg5\") pod \"cert-manager-webhook-5dbbbf75cc-brtbn\" (UID: \"f94f4767-3f84-429a-bff5-38c4fa82deaa\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-brtbn" Nov 01 05:51:36.711692 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:36.711610 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-brtbn" Nov 01 05:51:37.083630 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:37.083602 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-brtbn"] Nov 01 05:51:37.085070 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:51:37.085040 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf94f4767_3f84_429a_bff5_38c4fa82deaa.slice/crio-3542c3ea6ea9ccd62ed89f875504d9fd21a94a83d88f1dc688a3cb1da66a5967 WatchSource:0}: Error finding container 3542c3ea6ea9ccd62ed89f875504d9fd21a94a83d88f1dc688a3cb1da66a5967: Status 404 returned error can't find the container with id 3542c3ea6ea9ccd62ed89f875504d9fd21a94a83d88f1dc688a3cb1da66a5967 Nov 01 05:51:37.446732 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:37.446650 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-brtbn" event={"ID":"f94f4767-3f84-429a-bff5-38c4fa82deaa","Type":"ContainerStarted","Data":"3542c3ea6ea9ccd62ed89f875504d9fd21a94a83d88f1dc688a3cb1da66a5967"} Nov 01 05:51:39.980194 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:51:39.980139 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 05:51:40.410934 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:40.410902 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6c4qz" Nov 01 05:51:41.469804 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:41.469771 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-brtbn" event={"ID":"f94f4767-3f84-429a-bff5-38c4fa82deaa","Type":"ContainerStarted","Data":"fe230092a4f07b9ec0adac113d1eca27f8a843c76e881063d5d4e0302c40c34e"} Nov 01 05:51:41.470222 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:41.469907 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-brtbn" Nov 01 05:51:41.588719 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:41.588667 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-brtbn" podStartSLOduration=1.700023456 podStartE2EDuration="5.58865548s" totalImagesPullingTime="3.888632024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:51:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:51:41.588323007 +0000 UTC m=+380.243440176" watchObservedRunningTime="2026-11-01 05:51:41.58865548 +0000 UTC m=+380.243772646" Nov 01 05:51:47.479769 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:47.479735 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-brtbn" Nov 01 05:51:50.243110 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:50.243074 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-84b94dcd67-jsghm"] Nov 01 05:51:50.253311 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:50.253278 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84b94dcd67-jsghm" Nov 01 05:51:50.281266 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:50.281215 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tkq2\" (UniqueName: \"kubernetes.io/projected/5e90e7f9-4509-484e-be03-f470072753c7-kube-api-access-4tkq2\") pod \"kube-auth-proxy-84b94dcd67-jsghm\" (UID: \"5e90e7f9-4509-484e-be03-f470072753c7\") " pod="openshift-ingress/kube-auth-proxy-84b94dcd67-jsghm" Nov 01 05:51:50.281454 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:50.281280 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e90e7f9-4509-484e-be03-f470072753c7-tmp\") pod \"kube-auth-proxy-84b94dcd67-jsghm\" (UID: \"5e90e7f9-4509-484e-be03-f470072753c7\") " pod="openshift-ingress/kube-auth-proxy-84b94dcd67-jsghm" Nov 01 05:51:50.281549 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:50.281447 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e90e7f9-4509-484e-be03-f470072753c7-tls-certs\") pod \"kube-auth-proxy-84b94dcd67-jsghm\" (UID: \"5e90e7f9-4509-484e-be03-f470072753c7\") " pod="openshift-ingress/kube-auth-proxy-84b94dcd67-jsghm" Nov 01 05:51:50.316430 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:50.316396 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 01 05:51:50.357392 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:50.357357 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-fqlfv\"" Nov 01 05:51:50.357392 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:50.357394 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 01 05:51:50.359745 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:50.359721 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84b94dcd67-jsghm"] Nov 01 05:51:50.382176 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:50.382143 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e90e7f9-4509-484e-be03-f470072753c7-tls-certs\") pod \"kube-auth-proxy-84b94dcd67-jsghm\" (UID: \"5e90e7f9-4509-484e-be03-f470072753c7\") " pod="openshift-ingress/kube-auth-proxy-84b94dcd67-jsghm" Nov 01 05:51:50.382363 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:50.382186 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tkq2\" (UniqueName: \"kubernetes.io/projected/5e90e7f9-4509-484e-be03-f470072753c7-kube-api-access-4tkq2\") pod \"kube-auth-proxy-84b94dcd67-jsghm\" (UID: \"5e90e7f9-4509-484e-be03-f470072753c7\") " pod="openshift-ingress/kube-auth-proxy-84b94dcd67-jsghm" Nov 01 05:51:50.382363 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:50.382230 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e90e7f9-4509-484e-be03-f470072753c7-tmp\") pod \"kube-auth-proxy-84b94dcd67-jsghm\" (UID: \"5e90e7f9-4509-484e-be03-f470072753c7\") " pod="openshift-ingress/kube-auth-proxy-84b94dcd67-jsghm" Nov 01 05:51:50.384618 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:50.384583 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e90e7f9-4509-484e-be03-f470072753c7-tmp\") pod \"kube-auth-proxy-84b94dcd67-jsghm\" (UID: \"5e90e7f9-4509-484e-be03-f470072753c7\") " pod="openshift-ingress/kube-auth-proxy-84b94dcd67-jsghm" Nov 01 05:51:50.384812 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:50.384796 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e90e7f9-4509-484e-be03-f470072753c7-tls-certs\") pod \"kube-auth-proxy-84b94dcd67-jsghm\" (UID: \"5e90e7f9-4509-484e-be03-f470072753c7\") " pod="openshift-ingress/kube-auth-proxy-84b94dcd67-jsghm" Nov 01 05:51:50.449535 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:50.449503 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tkq2\" (UniqueName: \"kubernetes.io/projected/5e90e7f9-4509-484e-be03-f470072753c7-kube-api-access-4tkq2\") pod \"kube-auth-proxy-84b94dcd67-jsghm\" (UID: \"5e90e7f9-4509-484e-be03-f470072753c7\") " pod="openshift-ingress/kube-auth-proxy-84b94dcd67-jsghm" Nov 01 05:51:50.566055 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:50.566018 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84b94dcd67-jsghm" Nov 01 05:51:50.941192 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:50.941166 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84b94dcd67-jsghm"] Nov 01 05:51:50.942989 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:51:50.942956 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e90e7f9_4509_484e_be03_f470072753c7.slice/crio-54fc39e3c21e35da0ee295cec157e49df7797313af189e9d5809b4e7d0ac19f5 WatchSource:0}: Error finding container 54fc39e3c21e35da0ee295cec157e49df7797313af189e9d5809b4e7d0ac19f5: Status 404 returned error can't find the container with id 54fc39e3c21e35da0ee295cec157e49df7797313af189e9d5809b4e7d0ac19f5 Nov 01 05:51:51.520900 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:51.520868 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84b94dcd67-jsghm" event={"ID":"5e90e7f9-4509-484e-be03-f470072753c7","Type":"ContainerStarted","Data":"54fc39e3c21e35da0ee295cec157e49df7797313af189e9d5809b4e7d0ac19f5"} Nov 01 05:51:54.546024 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:54.545988 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84b94dcd67-jsghm" event={"ID":"5e90e7f9-4509-484e-be03-f470072753c7","Type":"ContainerStarted","Data":"8f9dc1779363d0f1e0581b88f198884d730e512b8f66f7625d5293a998b230e0"} Nov 01 05:51:54.664950 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:54.664887 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-84b94dcd67-jsghm" podStartSLOduration=1.784084757 podStartE2EDuration="4.664874683s" totalImagesPullingTime="2.880789926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:51:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:51:54.663766542 +0000 UTC m=+393.318883705" watchObservedRunningTime="2026-11-01 05:51:54.664874683 +0000 UTC m=+393.319991849" Nov 01 05:51:57.270330 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:57.270293 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-4xc9h"] Nov 01 05:51:57.276132 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:57.276111 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-4xc9h" Nov 01 05:51:57.329539 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:57.329502 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-4xc9h"] Nov 01 05:51:57.343806 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:57.343776 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4w9l\" (UniqueName: \"kubernetes.io/projected/e37a07a0-3cb2-4e6c-b720-5d9ca9ef19b2-kube-api-access-b4w9l\") pod \"authorino-56c6c9ddd5-4xc9h\" (UID: \"e37a07a0-3cb2-4e6c-b720-5d9ca9ef19b2\") " pod="kuadrant-system/authorino-56c6c9ddd5-4xc9h" Nov 01 05:51:57.378148 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:57.378116 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wgl92\"" Nov 01 05:51:57.444860 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:57.444824 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4w9l\" (UniqueName: \"kubernetes.io/projected/e37a07a0-3cb2-4e6c-b720-5d9ca9ef19b2-kube-api-access-b4w9l\") pod \"authorino-56c6c9ddd5-4xc9h\" (UID: \"e37a07a0-3cb2-4e6c-b720-5d9ca9ef19b2\") " pod="kuadrant-system/authorino-56c6c9ddd5-4xc9h" Nov 01 05:51:57.506948 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:57.506912 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4w9l\" (UniqueName: \"kubernetes.io/projected/e37a07a0-3cb2-4e6c-b720-5d9ca9ef19b2-kube-api-access-b4w9l\") pod \"authorino-56c6c9ddd5-4xc9h\" (UID: \"e37a07a0-3cb2-4e6c-b720-5d9ca9ef19b2\") " pod="kuadrant-system/authorino-56c6c9ddd5-4xc9h" Nov 01 05:51:57.588738 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:57.588664 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-4xc9h" Nov 01 05:51:57.957928 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:57.957899 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-4xc9h"] Nov 01 05:51:57.959574 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:51:57.959538 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode37a07a0_3cb2_4e6c_b720_5d9ca9ef19b2.slice/crio-7c1c9587e2bb704170d5339874f52c7ebcce9a132c02e53233d0a0f7e65c7efe WatchSource:0}: Error finding container 7c1c9587e2bb704170d5339874f52c7ebcce9a132c02e53233d0a0f7e65c7efe: Status 404 returned error can't find the container with id 7c1c9587e2bb704170d5339874f52c7ebcce9a132c02e53233d0a0f7e65c7efe Nov 01 05:51:58.565943 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:51:58.565906 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-4xc9h" event={"ID":"e37a07a0-3cb2-4e6c-b720-5d9ca9ef19b2","Type":"ContainerStarted","Data":"7c1c9587e2bb704170d5339874f52c7ebcce9a132c02e53233d0a0f7e65c7efe"} Nov 01 05:52:00.578594 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:52:00.578550 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-4xc9h" event={"ID":"e37a07a0-3cb2-4e6c-b720-5d9ca9ef19b2","Type":"ContainerStarted","Data":"6c3eafdc853cc2a52efac4a1b62f206d4e7fa92699b8fc9874ce9b54f6d42c8e"} Nov 01 05:52:00.694059 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:52:00.693964 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-4xc9h" podStartSLOduration=1.191893124 podStartE2EDuration="3.693950876s" totalImagesPullingTime="2.502057752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:51:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:52:00.693178564 +0000 UTC m=+399.348295731" watchObservedRunningTime="2026-11-01 05:52:00.693950876 +0000 UTC m=+399.349068042" Nov 01 05:52:09.971805 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:52:09.971759 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 05:52:39.972229 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:52:39.972190 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 05:52:47.914150 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:52:47.914106 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x2lbs"] Nov 01 05:52:47.918135 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:52:47.918110 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x2lbs" Nov 01 05:52:47.973499 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:52:47.973439 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 01 05:52:47.973678 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:52:47.973499 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 01 05:52:47.974885 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:52:47.974859 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1edd6e44-240f-4f06-9294-f04fe4fcda65-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x2lbs\" (UID: \"1edd6e44-240f-4f06-9294-f04fe4fcda65\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x2lbs" Nov 01 05:52:47.975015 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:52:47.974901 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ndq7\" (UniqueName: \"kubernetes.io/projected/1edd6e44-240f-4f06-9294-f04fe4fcda65-kube-api-access-6ndq7\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x2lbs\" (UID: \"1edd6e44-240f-4f06-9294-f04fe4fcda65\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x2lbs" Nov 01 05:52:47.975015 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:52:47.974925 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/1edd6e44-240f-4f06-9294-f04fe4fcda65-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x2lbs\" (UID: \"1edd6e44-240f-4f06-9294-f04fe4fcda65\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x2lbs" Nov 01 05:52:47.975110 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:52:47.975087 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1edd6e44-240f-4f06-9294-f04fe4fcda65-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x2lbs\" (UID: \"1edd6e44-240f-4f06-9294-f04fe4fcda65\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x2lbs" Nov 01 05:52:48.024156 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:52:48.024112 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 01 05:52:48.024156 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:52:48.024159 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 01 05:52:48.024579 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:52:48.024552 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 01 05:52:48.024702 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:52:48.024626 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-8fgch\"" Nov 01 05:52:48.026355 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:52:48.026334 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x2lbs"] Nov 01 05:52:48.075989 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:52:48.075961 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ndq7\" (UniqueName: \"kubernetes.io/projected/1edd6e44-240f-4f06-9294-f04fe4fcda65-kube-api-access-6ndq7\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x2lbs\" (UID: \"1edd6e44-240f-4f06-9294-f04fe4fcda65\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x2lbs" Nov 01 05:52:48.076149 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:52:48.076007 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/1edd6e44-240f-4f06-9294-f04fe4fcda65-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x2lbs\" (UID: \"1edd6e44-240f-4f06-9294-f04fe4fcda65\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x2lbs" Nov 01 05:52:48.076149 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:52:48.076062 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1edd6e44-240f-4f06-9294-f04fe4fcda65-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x2lbs\" (UID: \"1edd6e44-240f-4f06-9294-f04fe4fcda65\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x2lbs" Nov 01 05:52:48.076276 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:52:48.076168 5040 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Nov 01 05:52:48.076276 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:52:48.076219 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1edd6e44-240f-4f06-9294-f04fe4fcda65-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x2lbs\" (UID: \"1edd6e44-240f-4f06-9294-f04fe4fcda65\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x2lbs" Nov 01 05:52:48.076276 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:52:48.076252 5040 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1edd6e44-240f-4f06-9294-f04fe4fcda65-proxy-source podName:1edd6e44-240f-4f06-9294-f04fe4fcda65 nodeName:}" failed. No retries permitted until 2026-11-01 05:52:48.576225259 +0000 UTC m=+447.231342424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/1edd6e44-240f-4f06-9294-f04fe4fcda65-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-x2lbs" (UID: "1edd6e44-240f-4f06-9294-f04fe4fcda65") : configmap "usage-tenancy-proxy-source" not found Nov 01 05:52:48.076448 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:52:48.076308 5040 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Nov 01 05:52:48.076448 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:52:48.076365 5040 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1edd6e44-240f-4f06-9294-f04fe4fcda65-tls-cert podName:1edd6e44-240f-4f06-9294-f04fe4fcda65 nodeName:}" failed. No retries permitted until 2026-11-01 05:52:48.576350139 +0000 UTC m=+447.231467304 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/1edd6e44-240f-4f06-9294-f04fe4fcda65-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-x2lbs" (UID: "1edd6e44-240f-4f06-9294-f04fe4fcda65") : secret "usage-logs-tenancy-proxy-tls" not found Nov 01 05:52:48.076853 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:52:48.076830 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1edd6e44-240f-4f06-9294-f04fe4fcda65-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x2lbs\" (UID: \"1edd6e44-240f-4f06-9294-f04fe4fcda65\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x2lbs" Nov 01 05:52:48.140289 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:52:48.140252 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ndq7\" (UniqueName: \"kubernetes.io/projected/1edd6e44-240f-4f06-9294-f04fe4fcda65-kube-api-access-6ndq7\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x2lbs\" (UID: \"1edd6e44-240f-4f06-9294-f04fe4fcda65\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x2lbs" Nov 01 05:52:48.581067 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:52:48.581028 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1edd6e44-240f-4f06-9294-f04fe4fcda65-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x2lbs\" (UID: \"1edd6e44-240f-4f06-9294-f04fe4fcda65\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x2lbs" Nov 01 05:52:48.581251 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:52:48.581080 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/1edd6e44-240f-4f06-9294-f04fe4fcda65-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x2lbs\" (UID: \"1edd6e44-240f-4f06-9294-f04fe4fcda65\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x2lbs" Nov 01 05:52:48.582561 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:52:48.582528 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/1edd6e44-240f-4f06-9294-f04fe4fcda65-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x2lbs\" (UID: \"1edd6e44-240f-4f06-9294-f04fe4fcda65\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x2lbs" Nov 01 05:52:48.588620 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:52:48.588593 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1edd6e44-240f-4f06-9294-f04fe4fcda65-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x2lbs\" (UID: \"1edd6e44-240f-4f06-9294-f04fe4fcda65\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x2lbs" Nov 01 05:52:48.830389 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:52:48.830355 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x2lbs" Nov 01 05:52:49.206352 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:52:49.206326 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x2lbs"] Nov 01 05:52:49.207639 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:52:49.207606 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1edd6e44_240f_4f06_9294_f04fe4fcda65.slice/crio-60b93eb9ee21a8118ae3aeb5b65296ada50dcff8f334f178c73eac3bbe1dea3b WatchSource:0}: Error finding container 60b93eb9ee21a8118ae3aeb5b65296ada50dcff8f334f178c73eac3bbe1dea3b: Status 404 returned error can't find the container with id 60b93eb9ee21a8118ae3aeb5b65296ada50dcff8f334f178c73eac3bbe1dea3b Nov 01 05:52:49.825200 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:52:49.825164 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x2lbs" event={"ID":"1edd6e44-240f-4f06-9294-f04fe4fcda65","Type":"ContainerStarted","Data":"60b93eb9ee21a8118ae3aeb5b65296ada50dcff8f334f178c73eac3bbe1dea3b"} Nov 01 05:53:01.898948 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:01.898907 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x2lbs" event={"ID":"1edd6e44-240f-4f06-9294-f04fe4fcda65","Type":"ContainerStarted","Data":"48246e868458330b0d15280905271c36a57ddf58d114c79efe0d5a12c2423570"} Nov 01 05:53:01.899312 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:01.899051 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x2lbs" Nov 01 05:53:02.018498 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:02.018427 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x2lbs" podStartSLOduration=2.407964576 podStartE2EDuration="15.018410963s" totalImagesPullingTime="12.610446387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:52:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:53:02.016563495 +0000 UTC m=+460.671680663" watchObservedRunningTime="2026-11-01 05:53:02.018410963 +0000 UTC m=+460.673528129" Nov 01 05:53:09.972541 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:53:09.972500 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 05:53:10.132793 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:10.132759 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-zwgwr"] Nov 01 05:53:10.136144 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:10.136123 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zwgwr" Nov 01 05:53:10.242671 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:10.242596 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 01 05:53:10.242671 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:10.242646 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-gs9j9\"" Nov 01 05:53:10.244837 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:10.244814 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-zwgwr"] Nov 01 05:53:10.296114 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:10.296083 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbz48\" (UniqueName: \"kubernetes.io/projected/a5aa7cba-3da3-4424-a24f-0fcca53f4728-kube-api-access-jbz48\") pod \"llmisvc-controller-manager-869d5dcc5d-zwgwr\" (UID: \"a5aa7cba-3da3-4424-a24f-0fcca53f4728\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zwgwr" Nov 01 05:53:10.296257 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:10.296132 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a5aa7cba-3da3-4424-a24f-0fcca53f4728-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-zwgwr\" (UID: \"a5aa7cba-3da3-4424-a24f-0fcca53f4728\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zwgwr" Nov 01 05:53:10.358510 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:10.358462 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-c24v5"] Nov 01 05:53:10.362092 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:10.362070 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-c24v5" Nov 01 05:53:10.396622 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:10.396589 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbz48\" (UniqueName: \"kubernetes.io/projected/a5aa7cba-3da3-4424-a24f-0fcca53f4728-kube-api-access-jbz48\") pod \"llmisvc-controller-manager-869d5dcc5d-zwgwr\" (UID: \"a5aa7cba-3da3-4424-a24f-0fcca53f4728\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zwgwr" Nov 01 05:53:10.396771 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:10.396636 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a5aa7cba-3da3-4424-a24f-0fcca53f4728-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-zwgwr\" (UID: \"a5aa7cba-3da3-4424-a24f-0fcca53f4728\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zwgwr" Nov 01 05:53:10.398972 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:10.398953 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a5aa7cba-3da3-4424-a24f-0fcca53f4728-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-zwgwr\" (UID: \"a5aa7cba-3da3-4424-a24f-0fcca53f4728\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zwgwr" Nov 01 05:53:10.417729 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:10.417702 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 01 05:53:10.417889 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:10.417765 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 01 05:53:10.417952 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:10.417770 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-nchcg\"" Nov 01 05:53:10.420182 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:10.420158 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-c24v5"] Nov 01 05:53:10.458549 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:10.458525 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbz48\" (UniqueName: \"kubernetes.io/projected/a5aa7cba-3da3-4424-a24f-0fcca53f4728-kube-api-access-jbz48\") pod \"llmisvc-controller-manager-869d5dcc5d-zwgwr\" (UID: \"a5aa7cba-3da3-4424-a24f-0fcca53f4728\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zwgwr" Nov 01 05:53:10.497512 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:10.497446 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d73d9f29-4a48-46bf-8352-868e7d913a1b-cert\") pod \"odh-model-controller-769875db56-c24v5\" (UID: \"d73d9f29-4a48-46bf-8352-868e7d913a1b\") " pod="opendatahub/odh-model-controller-769875db56-c24v5" Nov 01 05:53:10.497664 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:10.497517 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdjxh\" (UniqueName: \"kubernetes.io/projected/d73d9f29-4a48-46bf-8352-868e7d913a1b-kube-api-access-cdjxh\") pod \"odh-model-controller-769875db56-c24v5\" (UID: \"d73d9f29-4a48-46bf-8352-868e7d913a1b\") " pod="opendatahub/odh-model-controller-769875db56-c24v5" Nov 01 05:53:10.497664 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:10.497561 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d73d9f29-4a48-46bf-8352-868e7d913a1b-metrics-certs\") pod \"odh-model-controller-769875db56-c24v5\" (UID: \"d73d9f29-4a48-46bf-8352-868e7d913a1b\") " pod="opendatahub/odh-model-controller-769875db56-c24v5" Nov 01 05:53:10.598445 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:10.598412 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d73d9f29-4a48-46bf-8352-868e7d913a1b-cert\") pod \"odh-model-controller-769875db56-c24v5\" (UID: \"d73d9f29-4a48-46bf-8352-868e7d913a1b\") " pod="opendatahub/odh-model-controller-769875db56-c24v5" Nov 01 05:53:10.598631 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:10.598456 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdjxh\" (UniqueName: \"kubernetes.io/projected/d73d9f29-4a48-46bf-8352-868e7d913a1b-kube-api-access-cdjxh\") pod \"odh-model-controller-769875db56-c24v5\" (UID: \"d73d9f29-4a48-46bf-8352-868e7d913a1b\") " pod="opendatahub/odh-model-controller-769875db56-c24v5" Nov 01 05:53:10.598631 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:10.598475 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d73d9f29-4a48-46bf-8352-868e7d913a1b-metrics-certs\") pod \"odh-model-controller-769875db56-c24v5\" (UID: \"d73d9f29-4a48-46bf-8352-868e7d913a1b\") " pod="opendatahub/odh-model-controller-769875db56-c24v5" Nov 01 05:53:10.600956 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:10.600932 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d73d9f29-4a48-46bf-8352-868e7d913a1b-metrics-certs\") pod \"odh-model-controller-769875db56-c24v5\" (UID: \"d73d9f29-4a48-46bf-8352-868e7d913a1b\") " pod="opendatahub/odh-model-controller-769875db56-c24v5" Nov 01 05:53:10.601034 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:10.600954 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d73d9f29-4a48-46bf-8352-868e7d913a1b-cert\") pod \"odh-model-controller-769875db56-c24v5\" (UID: \"d73d9f29-4a48-46bf-8352-868e7d913a1b\") " pod="opendatahub/odh-model-controller-769875db56-c24v5" Nov 01 05:53:10.660363 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:10.660329 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdjxh\" (UniqueName: \"kubernetes.io/projected/d73d9f29-4a48-46bf-8352-868e7d913a1b-kube-api-access-cdjxh\") pod \"odh-model-controller-769875db56-c24v5\" (UID: \"d73d9f29-4a48-46bf-8352-868e7d913a1b\") " pod="opendatahub/odh-model-controller-769875db56-c24v5" Nov 01 05:53:10.674190 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:10.674165 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-c24v5" Nov 01 05:53:10.748179 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:10.748116 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zwgwr" Nov 01 05:53:11.045433 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:11.045407 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-c24v5"] Nov 01 05:53:11.046833 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:53:11.046805 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd73d9f29_4a48_46bf_8352_868e7d913a1b.slice/crio-4d70de75672bbb80b6ee68f0033d6af7ece8e7386e80e2cdacded0ad2e237f0f WatchSource:0}: Error finding container 4d70de75672bbb80b6ee68f0033d6af7ece8e7386e80e2cdacded0ad2e237f0f: Status 404 returned error can't find the container with id 4d70de75672bbb80b6ee68f0033d6af7ece8e7386e80e2cdacded0ad2e237f0f Nov 01 05:53:11.156550 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:11.156526 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-zwgwr"] Nov 01 05:53:11.158232 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:53:11.158201 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda5aa7cba_3da3_4424_a24f_0fcca53f4728.slice/crio-5f7fc42e553654d732df47c2d427a641889101fb746f63fb60376ed3d4d977ca WatchSource:0}: Error finding container 5f7fc42e553654d732df47c2d427a641889101fb746f63fb60376ed3d4d977ca: Status 404 returned error can't find the container with id 5f7fc42e553654d732df47c2d427a641889101fb746f63fb60376ed3d4d977ca Nov 01 05:53:11.953845 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:11.953806 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zwgwr" event={"ID":"a5aa7cba-3da3-4424-a24f-0fcca53f4728","Type":"ContainerStarted","Data":"5f7fc42e553654d732df47c2d427a641889101fb746f63fb60376ed3d4d977ca"} Nov 01 05:53:11.955181 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:11.955148 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-c24v5" event={"ID":"d73d9f29-4a48-46bf-8352-868e7d913a1b","Type":"ContainerStarted","Data":"4d70de75672bbb80b6ee68f0033d6af7ece8e7386e80e2cdacded0ad2e237f0f"} Nov 01 05:53:12.911996 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:12.911961 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x2lbs" Nov 01 05:53:14.975074 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:14.975037 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-c24v5" event={"ID":"d73d9f29-4a48-46bf-8352-868e7d913a1b","Type":"ContainerStarted","Data":"d419a3dc03d1112a03341e2b7192c0b273ec246562937bc30992dda920ca464b"} Nov 01 05:53:14.975593 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:14.975105 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-c24v5" Nov 01 05:53:14.976725 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:14.976697 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zwgwr" event={"ID":"a5aa7cba-3da3-4424-a24f-0fcca53f4728","Type":"ContainerStarted","Data":"8fc6f89716bbad238142e3e9c8d57312aa896b154f1f9a9ae1bc72675f52091f"} Nov 01 05:53:14.976825 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:14.976777 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zwgwr" Nov 01 05:53:15.093698 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:15.093643 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-c24v5" podStartSLOduration=1.834600332 podStartE2EDuration="5.093629379s" totalImagesPullingTime="3.259029047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:53:15.092783812 +0000 UTC m=+473.747900978" watchObservedRunningTime="2026-11-01 05:53:15.093629379 +0000 UTC m=+473.748746546" Nov 01 05:53:15.211259 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:15.211200 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zwgwr" podStartSLOduration=1.955405815 podStartE2EDuration="5.211185312s" totalImagesPullingTime="3.255779497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:53:15.210015559 +0000 UTC m=+473.865132725" watchObservedRunningTime="2026-11-01 05:53:15.211185312 +0000 UTC m=+473.866302517" Nov 01 05:53:25.985229 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:25.985193 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-c24v5" Nov 01 05:53:35.595757 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:35.595705 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-g9wcq"] Nov 01 05:53:35.598473 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:35.598447 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-g9wcq" Nov 01 05:53:35.666490 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:35.666438 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-g9wcq"] Nov 01 05:53:35.712296 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:35.712244 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mvfs\" (UniqueName: \"kubernetes.io/projected/bffd48c1-c326-4b40-8b63-f9c6aaebd8e4-kube-api-access-4mvfs\") pod \"authorino-6f6944bcdd-g9wcq\" (UID: \"bffd48c1-c326-4b40-8b63-f9c6aaebd8e4\") " pod="kuadrant-system/authorino-6f6944bcdd-g9wcq" Nov 01 05:53:35.813071 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:35.813031 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mvfs\" (UniqueName: \"kubernetes.io/projected/bffd48c1-c326-4b40-8b63-f9c6aaebd8e4-kube-api-access-4mvfs\") pod \"authorino-6f6944bcdd-g9wcq\" (UID: \"bffd48c1-c326-4b40-8b63-f9c6aaebd8e4\") " pod="kuadrant-system/authorino-6f6944bcdd-g9wcq" Nov 01 05:53:35.875530 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:35.875449 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mvfs\" (UniqueName: \"kubernetes.io/projected/bffd48c1-c326-4b40-8b63-f9c6aaebd8e4-kube-api-access-4mvfs\") pod \"authorino-6f6944bcdd-g9wcq\" (UID: \"bffd48c1-c326-4b40-8b63-f9c6aaebd8e4\") " pod="kuadrant-system/authorino-6f6944bcdd-g9wcq" Nov 01 05:53:35.909345 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:35.909315 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-g9wcq" Nov 01 05:53:35.928361 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:35.928330 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-g9wcq"] Nov 01 05:53:35.978030 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:35.977993 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c59d664f9-p5ckv"] Nov 01 05:53:35.983533 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:35.981322 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c59d664f9-p5ckv" Nov 01 05:53:36.014772 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:36.014745 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9p2w\" (UniqueName: \"kubernetes.io/projected/66b689ab-8b85-41a7-a652-a5a29d7ef7db-kube-api-access-h9p2w\") pod \"authorino-c59d664f9-p5ckv\" (UID: \"66b689ab-8b85-41a7-a652-a5a29d7ef7db\") " pod="kuadrant-system/authorino-c59d664f9-p5ckv" Nov 01 05:53:36.021092 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:36.021066 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c59d664f9-p5ckv"] Nov 01 05:53:36.085303 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:36.085232 5040 generic.go:356] "Generic (PLEG): container finished" podID="baa43c0f-ec86-4262-9bdc-773e9b11bdbb" containerID="2ef15770562fae9547d3fdcef5b63227b8bef2b9218426060ac98e29e6f41618" exitCode=0 Nov 01 05:53:36.085303 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:36.085302 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" event={"ID":"baa43c0f-ec86-4262-9bdc-773e9b11bdbb","Type":"ContainerDied","Data":"2ef15770562fae9547d3fdcef5b63227b8bef2b9218426060ac98e29e6f41618"} Nov 01 05:53:36.085687 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:36.085672 5040 scope.go:122] "RemoveContainer" containerID="2ef15770562fae9547d3fdcef5b63227b8bef2b9218426060ac98e29e6f41618" Nov 01 05:53:36.086785 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:36.086755 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 05:53:36.086853 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:36.086841 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 05:53:36.086884 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:36.086872 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 05:53:36.115241 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:36.115214 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9p2w\" (UniqueName: \"kubernetes.io/projected/66b689ab-8b85-41a7-a652-a5a29d7ef7db-kube-api-access-h9p2w\") pod \"authorino-c59d664f9-p5ckv\" (UID: \"66b689ab-8b85-41a7-a652-a5a29d7ef7db\") " pod="kuadrant-system/authorino-c59d664f9-p5ckv" Nov 01 05:53:36.177348 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:36.177311 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9p2w\" (UniqueName: \"kubernetes.io/projected/66b689ab-8b85-41a7-a652-a5a29d7ef7db-kube-api-access-h9p2w\") pod \"authorino-c59d664f9-p5ckv\" (UID: \"66b689ab-8b85-41a7-a652-a5a29d7ef7db\") " pod="kuadrant-system/authorino-c59d664f9-p5ckv" Nov 01 05:53:36.315640 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:36.315609 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c59d664f9-p5ckv" Nov 01 05:53:36.332221 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:36.332189 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-g9wcq"] Nov 01 05:53:36.332708 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:53:36.332673 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbffd48c1_c326_4b40_8b63_f9c6aaebd8e4.slice/crio-1f7a4bbb1f3f9bb5a6e7a6b3e03e16828a71595494897b5b983d930e842bc831 WatchSource:0}: Error finding container 1f7a4bbb1f3f9bb5a6e7a6b3e03e16828a71595494897b5b983d930e842bc831: Status 404 returned error can't find the container with id 1f7a4bbb1f3f9bb5a6e7a6b3e03e16828a71595494897b5b983d930e842bc831 Nov 01 05:53:36.431873 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:36.431794 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c59d664f9-p5ckv"] Nov 01 05:53:36.689734 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:36.689707 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c59d664f9-p5ckv"] Nov 01 05:53:36.691201 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:53:36.691170 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66b689ab_8b85_41a7_a652_a5a29d7ef7db.slice/crio-58954d9011877ba179486b61e7c71181f64f1590d12c5464c0a023b1489058ba WatchSource:0}: Error finding container 58954d9011877ba179486b61e7c71181f64f1590d12c5464c0a023b1489058ba: Status 404 returned error can't find the container with id 58954d9011877ba179486b61e7c71181f64f1590d12c5464c0a023b1489058ba Nov 01 05:53:37.092892 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:37.092858 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c59d664f9-p5ckv" event={"ID":"66b689ab-8b85-41a7-a652-a5a29d7ef7db","Type":"ContainerStarted","Data":"58954d9011877ba179486b61e7c71181f64f1590d12c5464c0a023b1489058ba"} Nov 01 05:53:37.120044 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:37.120005 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" event={"ID":"baa43c0f-ec86-4262-9bdc-773e9b11bdbb","Type":"ContainerStarted","Data":"70fd85aa5cc8403e134941f7c20cabed7f6e1c089e2995c2dcaf4da71f9eda82"} Nov 01 05:53:37.121614 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:37.121588 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-g9wcq" event={"ID":"bffd48c1-c326-4b40-8b63-f9c6aaebd8e4","Type":"ContainerStarted","Data":"1f7a4bbb1f3f9bb5a6e7a6b3e03e16828a71595494897b5b983d930e842bc831"} Nov 01 05:53:37.946885 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:37.946859 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 05:53:37.947211 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:37.947044 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-mgnkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Nov 01 05:53:37.947211 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:37.947085 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" podUID="baa43c0f-ec86-4262-9bdc-773e9b11bdbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Nov 01 05:53:38.131104 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:38.131075 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-g9wcq" event={"ID":"bffd48c1-c326-4b40-8b63-f9c6aaebd8e4","Type":"ContainerStarted","Data":"2998617acff2571261b2e39e1e95a20e669040b82b91118c3bf60dcb33396628"} Nov 01 05:53:38.131104 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:38.131093 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-g9wcq" podUID="bffd48c1-c326-4b40-8b63-f9c6aaebd8e4" containerName="authorino" containerID="cri-o://2998617acff2571261b2e39e1e95a20e669040b82b91118c3bf60dcb33396628" gracePeriod=30 Nov 01 05:53:38.246743 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:38.246681 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-g9wcq" podStartSLOduration=1.698289161 podStartE2EDuration="3.246663995s" totalImagesPullingTime="1.548374834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:53:38.246606335 +0000 UTC m=+496.901723501" watchObservedRunningTime="2026-11-01 05:53:38.246663995 +0000 UTC m=+496.901781169" Nov 01 05:53:38.424718 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:38.424686 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-g9wcq" Nov 01 05:53:38.439014 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:38.438984 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bffd48c1-c326-4b40-8b63-f9c6aaebd8e4-kube-api-access-4mvfs\" (UniqueName: \"kubernetes.io/projected/bffd48c1-c326-4b40-8b63-f9c6aaebd8e4-kube-api-access-4mvfs\") pod \"bffd48c1-c326-4b40-8b63-f9c6aaebd8e4\" (UID: \"bffd48c1-c326-4b40-8b63-f9c6aaebd8e4\") " Nov 01 05:53:38.441150 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:38.441121 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bffd48c1-c326-4b40-8b63-f9c6aaebd8e4-kube-api-access-4mvfs" pod "bffd48c1-c326-4b40-8b63-f9c6aaebd8e4" (UID: "bffd48c1-c326-4b40-8b63-f9c6aaebd8e4"). InnerVolumeSpecName "kube-api-access-4mvfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:53:38.539823 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:38.539788 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mvfs\" (UniqueName: \"kubernetes.io/projected/bffd48c1-c326-4b40-8b63-f9c6aaebd8e4-kube-api-access-4mvfs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:38.949503 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:38.947845 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-mgnkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Nov 01 05:53:38.949503 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:38.947896 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" podUID="baa43c0f-ec86-4262-9bdc-773e9b11bdbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Nov 01 05:53:39.139417 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:39.139378 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c59d664f9-p5ckv" event={"ID":"66b689ab-8b85-41a7-a652-a5a29d7ef7db","Type":"ContainerStarted","Data":"df7e82bf6fdda95dbf6819940bd7f934dbf03e4991a95e8ff66a39a3e876d2c2"} Nov 01 05:53:39.139625 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:39.139549 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-c59d664f9-p5ckv" podUID="66b689ab-8b85-41a7-a652-a5a29d7ef7db" containerName="authorino" containerID="cri-o://df7e82bf6fdda95dbf6819940bd7f934dbf03e4991a95e8ff66a39a3e876d2c2" gracePeriod=30 Nov 01 05:53:39.141434 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:39.141357 5040 generic.go:356] "Generic (PLEG): container finished" podID="bffd48c1-c326-4b40-8b63-f9c6aaebd8e4" containerID="2998617acff2571261b2e39e1e95a20e669040b82b91118c3bf60dcb33396628" exitCode=0 Nov 01 05:53:39.141983 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:39.141398 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-g9wcq" Nov 01 05:53:39.141983 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:39.141403 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-g9wcq" event={"ID":"bffd48c1-c326-4b40-8b63-f9c6aaebd8e4","Type":"ContainerDied","Data":"2998617acff2571261b2e39e1e95a20e669040b82b91118c3bf60dcb33396628"} Nov 01 05:53:39.141983 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:39.141793 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-g9wcq" event={"ID":"bffd48c1-c326-4b40-8b63-f9c6aaebd8e4","Type":"ContainerDied","Data":"1f7a4bbb1f3f9bb5a6e7a6b3e03e16828a71595494897b5b983d930e842bc831"} Nov 01 05:53:39.141983 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:39.141859 5040 scope.go:122] "RemoveContainer" containerID="2998617acff2571261b2e39e1e95a20e669040b82b91118c3bf60dcb33396628" Nov 01 05:53:39.160510 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:39.160472 5040 scope.go:122] "RemoveContainer" containerID="2998617acff2571261b2e39e1e95a20e669040b82b91118c3bf60dcb33396628" Nov 01 05:53:39.160851 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:53:39.160831 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2998617acff2571261b2e39e1e95a20e669040b82b91118c3bf60dcb33396628\": container with ID starting with 2998617acff2571261b2e39e1e95a20e669040b82b91118c3bf60dcb33396628 not found: ID does not exist" containerID="2998617acff2571261b2e39e1e95a20e669040b82b91118c3bf60dcb33396628" Nov 01 05:53:39.160912 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:39.160863 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2998617acff2571261b2e39e1e95a20e669040b82b91118c3bf60dcb33396628"} err="failed to get container status \"2998617acff2571261b2e39e1e95a20e669040b82b91118c3bf60dcb33396628\": rpc error: code = NotFound desc = could not find container \"2998617acff2571261b2e39e1e95a20e669040b82b91118c3bf60dcb33396628\": container with ID starting with 2998617acff2571261b2e39e1e95a20e669040b82b91118c3bf60dcb33396628 not found: ID does not exist" Nov 01 05:53:39.169551 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:39.169528 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-4xc9h"] Nov 01 05:53:39.169701 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:39.169682 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-4xc9h" podUID="e37a07a0-3cb2-4e6c-b720-5d9ca9ef19b2" containerName="authorino" containerID="cri-o://6c3eafdc853cc2a52efac4a1b62f206d4e7fa92699b8fc9874ce9b54f6d42c8e" gracePeriod=30 Nov 01 05:53:39.272258 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:39.272201 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-c59d664f9-p5ckv" podStartSLOduration=2.923042427 podStartE2EDuration="4.272185648s" totalImagesPullingTime="1.349143221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:53:39.270736929 +0000 UTC m=+497.925854097" watchObservedRunningTime="2026-11-01 05:53:39.272185648 +0000 UTC m=+497.927302816" Nov 01 05:53:39.477731 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:39.477702 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c59d664f9-p5ckv" Nov 01 05:53:39.480055 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:39.480038 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-4xc9h" Nov 01 05:53:39.528033 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:39.528005 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-g9wcq"] Nov 01 05:53:39.528033 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:39.528037 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-g9wcq"] Nov 01 05:53:39.548368 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:39.548339 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66b689ab-8b85-41a7-a652-a5a29d7ef7db-kube-api-access-h9p2w\" (UniqueName: \"kubernetes.io/projected/66b689ab-8b85-41a7-a652-a5a29d7ef7db-kube-api-access-h9p2w\") pod \"66b689ab-8b85-41a7-a652-a5a29d7ef7db\" (UID: \"66b689ab-8b85-41a7-a652-a5a29d7ef7db\") " Nov 01 05:53:39.548535 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:39.548386 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e37a07a0-3cb2-4e6c-b720-5d9ca9ef19b2-kube-api-access-b4w9l\" (UniqueName: \"kubernetes.io/projected/e37a07a0-3cb2-4e6c-b720-5d9ca9ef19b2-kube-api-access-b4w9l\") pod \"e37a07a0-3cb2-4e6c-b720-5d9ca9ef19b2\" (UID: \"e37a07a0-3cb2-4e6c-b720-5d9ca9ef19b2\") " Nov 01 05:53:39.550547 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:39.550517 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e37a07a0-3cb2-4e6c-b720-5d9ca9ef19b2-kube-api-access-b4w9l" pod "e37a07a0-3cb2-4e6c-b720-5d9ca9ef19b2" (UID: "e37a07a0-3cb2-4e6c-b720-5d9ca9ef19b2"). InnerVolumeSpecName "kube-api-access-b4w9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:53:39.550639 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:39.550571 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66b689ab-8b85-41a7-a652-a5a29d7ef7db-kube-api-access-h9p2w" pod "66b689ab-8b85-41a7-a652-a5a29d7ef7db" (UID: "66b689ab-8b85-41a7-a652-a5a29d7ef7db"). InnerVolumeSpecName "kube-api-access-h9p2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:53:39.649883 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:39.649849 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9p2w\" (UniqueName: \"kubernetes.io/projected/66b689ab-8b85-41a7-a652-a5a29d7ef7db-kube-api-access-h9p2w\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:39.649883 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:39.649877 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4w9l\" (UniqueName: \"kubernetes.io/projected/e37a07a0-3cb2-4e6c-b720-5d9ca9ef19b2-kube-api-access-b4w9l\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:39.947979 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:39.947941 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-mgnkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Nov 01 05:53:39.948177 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:39.948001 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" podUID="baa43c0f-ec86-4262-9bdc-773e9b11bdbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Nov 01 05:53:39.973032 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:53:39.972998 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 05:53:40.150839 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:40.150802 5040 generic.go:356] "Generic (PLEG): container finished" podID="66b689ab-8b85-41a7-a652-a5a29d7ef7db" containerID="df7e82bf6fdda95dbf6819940bd7f934dbf03e4991a95e8ff66a39a3e876d2c2" exitCode=0 Nov 01 05:53:40.151012 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:40.150849 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c59d664f9-p5ckv" Nov 01 05:53:40.151012 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:40.150884 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c59d664f9-p5ckv" event={"ID":"66b689ab-8b85-41a7-a652-a5a29d7ef7db","Type":"ContainerDied","Data":"df7e82bf6fdda95dbf6819940bd7f934dbf03e4991a95e8ff66a39a3e876d2c2"} Nov 01 05:53:40.151012 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:40.150932 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c59d664f9-p5ckv" event={"ID":"66b689ab-8b85-41a7-a652-a5a29d7ef7db","Type":"ContainerDied","Data":"58954d9011877ba179486b61e7c71181f64f1590d12c5464c0a023b1489058ba"} Nov 01 05:53:40.151012 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:40.150957 5040 scope.go:122] "RemoveContainer" containerID="df7e82bf6fdda95dbf6819940bd7f934dbf03e4991a95e8ff66a39a3e876d2c2" Nov 01 05:53:40.153652 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:40.153618 5040 generic.go:356] "Generic (PLEG): container finished" podID="e37a07a0-3cb2-4e6c-b720-5d9ca9ef19b2" containerID="6c3eafdc853cc2a52efac4a1b62f206d4e7fa92699b8fc9874ce9b54f6d42c8e" exitCode=0 Nov 01 05:53:40.153778 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:40.153645 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-4xc9h" event={"ID":"e37a07a0-3cb2-4e6c-b720-5d9ca9ef19b2","Type":"ContainerDied","Data":"6c3eafdc853cc2a52efac4a1b62f206d4e7fa92699b8fc9874ce9b54f6d42c8e"} Nov 01 05:53:40.153778 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:40.153678 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-4xc9h" Nov 01 05:53:40.153778 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:40.153692 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-4xc9h" event={"ID":"e37a07a0-3cb2-4e6c-b720-5d9ca9ef19b2","Type":"ContainerDied","Data":"7c1c9587e2bb704170d5339874f52c7ebcce9a132c02e53233d0a0f7e65c7efe"} Nov 01 05:53:40.162139 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:40.162119 5040 scope.go:122] "RemoveContainer" containerID="df7e82bf6fdda95dbf6819940bd7f934dbf03e4991a95e8ff66a39a3e876d2c2" Nov 01 05:53:40.162446 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:53:40.162426 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df7e82bf6fdda95dbf6819940bd7f934dbf03e4991a95e8ff66a39a3e876d2c2\": container with ID starting with df7e82bf6fdda95dbf6819940bd7f934dbf03e4991a95e8ff66a39a3e876d2c2 not found: ID does not exist" containerID="df7e82bf6fdda95dbf6819940bd7f934dbf03e4991a95e8ff66a39a3e876d2c2" Nov 01 05:53:40.162565 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:40.162458 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df7e82bf6fdda95dbf6819940bd7f934dbf03e4991a95e8ff66a39a3e876d2c2"} err="failed to get container status \"df7e82bf6fdda95dbf6819940bd7f934dbf03e4991a95e8ff66a39a3e876d2c2\": rpc error: code = NotFound desc = could not find container \"df7e82bf6fdda95dbf6819940bd7f934dbf03e4991a95e8ff66a39a3e876d2c2\": container with ID starting with df7e82bf6fdda95dbf6819940bd7f934dbf03e4991a95e8ff66a39a3e876d2c2 not found: ID does not exist" Nov 01 05:53:40.162565 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:40.162494 5040 scope.go:122] "RemoveContainer" containerID="6c3eafdc853cc2a52efac4a1b62f206d4e7fa92699b8fc9874ce9b54f6d42c8e" Nov 01 05:53:40.170644 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:40.170629 5040 scope.go:122] "RemoveContainer" containerID="6c3eafdc853cc2a52efac4a1b62f206d4e7fa92699b8fc9874ce9b54f6d42c8e" Nov 01 05:53:40.170898 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:53:40.170880 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c3eafdc853cc2a52efac4a1b62f206d4e7fa92699b8fc9874ce9b54f6d42c8e\": container with ID starting with 6c3eafdc853cc2a52efac4a1b62f206d4e7fa92699b8fc9874ce9b54f6d42c8e not found: ID does not exist" containerID="6c3eafdc853cc2a52efac4a1b62f206d4e7fa92699b8fc9874ce9b54f6d42c8e" Nov 01 05:53:40.170970 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:40.170903 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c3eafdc853cc2a52efac4a1b62f206d4e7fa92699b8fc9874ce9b54f6d42c8e"} err="failed to get container status \"6c3eafdc853cc2a52efac4a1b62f206d4e7fa92699b8fc9874ce9b54f6d42c8e\": rpc error: code = NotFound desc = could not find container \"6c3eafdc853cc2a52efac4a1b62f206d4e7fa92699b8fc9874ce9b54f6d42c8e\": container with ID starting with 6c3eafdc853cc2a52efac4a1b62f206d4e7fa92699b8fc9874ce9b54f6d42c8e not found: ID does not exist" Nov 01 05:53:40.331634 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:40.331588 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-4xc9h"] Nov 01 05:53:40.335627 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:40.335601 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-4xc9h"] Nov 01 05:53:40.446335 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:40.446298 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c59d664f9-p5ckv"] Nov 01 05:53:40.450029 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:40.450007 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c59d664f9-p5ckv"] Nov 01 05:53:40.947893 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:40.947857 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-mgnkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Nov 01 05:53:40.948044 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:40.947938 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" podUID="baa43c0f-ec86-4262-9bdc-773e9b11bdbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Nov 01 05:53:41.523158 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:41.523123 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66b689ab-8b85-41a7-a652-a5a29d7ef7db" path="/var/lib/kubelet/pods/66b689ab-8b85-41a7-a652-a5a29d7ef7db/volumes" Nov 01 05:53:41.523569 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:41.523439 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bffd48c1-c326-4b40-8b63-f9c6aaebd8e4" path="/var/lib/kubelet/pods/bffd48c1-c326-4b40-8b63-f9c6aaebd8e4/volumes" Nov 01 05:53:41.523752 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:41.523739 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e37a07a0-3cb2-4e6c-b720-5d9ca9ef19b2" path="/var/lib/kubelet/pods/e37a07a0-3cb2-4e6c-b720-5d9ca9ef19b2/volumes" Nov 01 05:53:41.947422 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:41.947343 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-mgnkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Nov 01 05:53:41.947422 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:41.947395 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" podUID="baa43c0f-ec86-4262-9bdc-773e9b11bdbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Nov 01 05:53:42.951757 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:42.951726 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 05:53:42.952188 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:42.952031 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 05:53:42.952805 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:42.952785 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 05:53:45.984780 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:53:45.984750 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zwgwr" Nov 01 05:54:09.971612 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:54:09.971576 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 05:54:20.451947 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:20.451911 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-788tv"] Nov 01 05:54:20.452333 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:20.452305 5040 cpu_manager.go:414] "containerMap: removing container" podUID="e37a07a0-3cb2-4e6c-b720-5d9ca9ef19b2" containerName="authorino" Nov 01 05:54:20.452333 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:20.452317 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e37a07a0-3cb2-4e6c-b720-5d9ca9ef19b2" containerName="authorino" Nov 01 05:54:20.452441 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:20.452349 5040 cpu_manager.go:414] "containerMap: removing container" podUID="bffd48c1-c326-4b40-8b63-f9c6aaebd8e4" containerName="authorino" Nov 01 05:54:20.452441 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:20.452354 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bffd48c1-c326-4b40-8b63-f9c6aaebd8e4" containerName="authorino" Nov 01 05:54:20.452441 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:20.452363 5040 cpu_manager.go:414] "containerMap: removing container" podUID="66b689ab-8b85-41a7-a652-a5a29d7ef7db" containerName="authorino" Nov 01 05:54:20.452441 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:20.452369 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66b689ab-8b85-41a7-a652-a5a29d7ef7db" containerName="authorino" Nov 01 05:54:20.452441 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:20.452430 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="e37a07a0-3cb2-4e6c-b720-5d9ca9ef19b2" containerName="authorino" Nov 01 05:54:20.452441 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:20.452443 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="bffd48c1-c326-4b40-8b63-f9c6aaebd8e4" containerName="authorino" Nov 01 05:54:20.452642 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:20.452449 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="66b689ab-8b85-41a7-a652-a5a29d7ef7db" containerName="authorino" Nov 01 05:54:20.454355 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:20.454340 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-788tv" Nov 01 05:54:20.510986 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:20.510955 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww5tn\" (UniqueName: \"kubernetes.io/projected/72a66fbc-a5fe-42f7-8362-fe8e2a62025f-kube-api-access-ww5tn\") pod \"llmisvc-controller-manager-6466c554d8-788tv\" (UID: \"72a66fbc-a5fe-42f7-8362-fe8e2a62025f\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-788tv" Nov 01 05:54:20.511156 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:20.511020 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/72a66fbc-a5fe-42f7-8362-fe8e2a62025f-cert\") pod \"llmisvc-controller-manager-6466c554d8-788tv\" (UID: \"72a66fbc-a5fe-42f7-8362-fe8e2a62025f\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-788tv" Nov 01 05:54:20.513208 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:20.513183 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-788tv"] Nov 01 05:54:20.611718 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:20.611679 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/72a66fbc-a5fe-42f7-8362-fe8e2a62025f-cert\") pod \"llmisvc-controller-manager-6466c554d8-788tv\" (UID: \"72a66fbc-a5fe-42f7-8362-fe8e2a62025f\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-788tv" Nov 01 05:54:20.611889 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:20.611773 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww5tn\" (UniqueName: \"kubernetes.io/projected/72a66fbc-a5fe-42f7-8362-fe8e2a62025f-kube-api-access-ww5tn\") pod \"llmisvc-controller-manager-6466c554d8-788tv\" (UID: \"72a66fbc-a5fe-42f7-8362-fe8e2a62025f\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-788tv" Nov 01 05:54:20.614245 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:20.614216 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/72a66fbc-a5fe-42f7-8362-fe8e2a62025f-cert\") pod \"llmisvc-controller-manager-6466c554d8-788tv\" (UID: \"72a66fbc-a5fe-42f7-8362-fe8e2a62025f\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-788tv" Nov 01 05:54:20.673724 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:20.673690 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww5tn\" (UniqueName: \"kubernetes.io/projected/72a66fbc-a5fe-42f7-8362-fe8e2a62025f-kube-api-access-ww5tn\") pod \"llmisvc-controller-manager-6466c554d8-788tv\" (UID: \"72a66fbc-a5fe-42f7-8362-fe8e2a62025f\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-788tv" Nov 01 05:54:20.765611 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:20.765577 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-788tv" Nov 01 05:54:21.135783 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:21.135752 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-788tv"] Nov 01 05:54:21.137174 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:54:21.137140 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72a66fbc_a5fe_42f7_8362_fe8e2a62025f.slice/crio-64451be8eb197c3e7555ee6d89df9a2cbada3a914e59f0f541fd9f911db77ea1 WatchSource:0}: Error finding container 64451be8eb197c3e7555ee6d89df9a2cbada3a914e59f0f541fd9f911db77ea1: Status 404 returned error can't find the container with id 64451be8eb197c3e7555ee6d89df9a2cbada3a914e59f0f541fd9f911db77ea1 Nov 01 05:54:21.358239 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:21.358200 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-788tv" event={"ID":"72a66fbc-a5fe-42f7-8362-fe8e2a62025f","Type":"ContainerStarted","Data":"64451be8eb197c3e7555ee6d89df9a2cbada3a914e59f0f541fd9f911db77ea1"} Nov 01 05:54:22.364497 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:22.364440 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-788tv" event={"ID":"72a66fbc-a5fe-42f7-8362-fe8e2a62025f","Type":"ContainerStarted","Data":"2452f40fc1b747b3d606b15ed5d6a59633332ea278dae2b65e7965f94388a8f5"} Nov 01 05:54:22.364936 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:22.364531 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-788tv" Nov 01 05:54:22.482766 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:22.482712 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-788tv" podStartSLOduration=1.783325009 podStartE2EDuration="2.482697478s" totalImagesPullingTime="699.372469ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:54:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:54:22.481972823 +0000 UTC m=+541.137089990" watchObservedRunningTime="2026-11-01 05:54:22.482697478 +0000 UTC m=+541.137814665" Nov 01 05:54:39.972708 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:54:39.972670 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 05:54:53.372179 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:53.372147 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-788tv" Nov 01 05:54:53.542057 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:53.542021 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-zwgwr"] Nov 01 05:54:53.542347 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:53.542288 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zwgwr" podUID="a5aa7cba-3da3-4424-a24f-0fcca53f4728" containerName="manager" containerID="cri-o://8fc6f89716bbad238142e3e9c8d57312aa896b154f1f9a9ae1bc72675f52091f" gracePeriod=30 Nov 01 05:54:53.788316 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:53.788290 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zwgwr" Nov 01 05:54:53.880340 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:53.880307 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5aa7cba-3da3-4424-a24f-0fcca53f4728-cert\" (UniqueName: \"kubernetes.io/secret/a5aa7cba-3da3-4424-a24f-0fcca53f4728-cert\") pod \"a5aa7cba-3da3-4424-a24f-0fcca53f4728\" (UID: \"a5aa7cba-3da3-4424-a24f-0fcca53f4728\") " Nov 01 05:54:53.880559 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:53.880351 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5aa7cba-3da3-4424-a24f-0fcca53f4728-kube-api-access-jbz48\" (UniqueName: \"kubernetes.io/projected/a5aa7cba-3da3-4424-a24f-0fcca53f4728-kube-api-access-jbz48\") pod \"a5aa7cba-3da3-4424-a24f-0fcca53f4728\" (UID: \"a5aa7cba-3da3-4424-a24f-0fcca53f4728\") " Nov 01 05:54:53.882589 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:53.882559 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5aa7cba-3da3-4424-a24f-0fcca53f4728-cert" pod "a5aa7cba-3da3-4424-a24f-0fcca53f4728" (UID: "a5aa7cba-3da3-4424-a24f-0fcca53f4728"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:54:53.882589 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:53.882562 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5aa7cba-3da3-4424-a24f-0fcca53f4728-kube-api-access-jbz48" pod "a5aa7cba-3da3-4424-a24f-0fcca53f4728" (UID: "a5aa7cba-3da3-4424-a24f-0fcca53f4728"). InnerVolumeSpecName "kube-api-access-jbz48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:54:53.981895 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:53.981819 5040 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a5aa7cba-3da3-4424-a24f-0fcca53f4728-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:54:53.981895 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:53.981849 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbz48\" (UniqueName: \"kubernetes.io/projected/a5aa7cba-3da3-4424-a24f-0fcca53f4728-kube-api-access-jbz48\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:54:54.523124 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:54.523089 5040 generic.go:356] "Generic (PLEG): container finished" podID="a5aa7cba-3da3-4424-a24f-0fcca53f4728" containerID="8fc6f89716bbad238142e3e9c8d57312aa896b154f1f9a9ae1bc72675f52091f" exitCode=0 Nov 01 05:54:54.523601 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:54.523141 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zwgwr" Nov 01 05:54:54.523601 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:54.523166 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zwgwr" event={"ID":"a5aa7cba-3da3-4424-a24f-0fcca53f4728","Type":"ContainerDied","Data":"8fc6f89716bbad238142e3e9c8d57312aa896b154f1f9a9ae1bc72675f52091f"} Nov 01 05:54:54.523601 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:54.523210 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zwgwr" event={"ID":"a5aa7cba-3da3-4424-a24f-0fcca53f4728","Type":"ContainerDied","Data":"5f7fc42e553654d732df47c2d427a641889101fb746f63fb60376ed3d4d977ca"} Nov 01 05:54:54.523601 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:54.523234 5040 scope.go:122] "RemoveContainer" containerID="8fc6f89716bbad238142e3e9c8d57312aa896b154f1f9a9ae1bc72675f52091f" Nov 01 05:54:54.532734 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:54.532717 5040 scope.go:122] "RemoveContainer" containerID="8fc6f89716bbad238142e3e9c8d57312aa896b154f1f9a9ae1bc72675f52091f" Nov 01 05:54:54.532993 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:54:54.532975 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fc6f89716bbad238142e3e9c8d57312aa896b154f1f9a9ae1bc72675f52091f\": container with ID starting with 8fc6f89716bbad238142e3e9c8d57312aa896b154f1f9a9ae1bc72675f52091f not found: ID does not exist" containerID="8fc6f89716bbad238142e3e9c8d57312aa896b154f1f9a9ae1bc72675f52091f" Nov 01 05:54:54.533060 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:54.533007 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fc6f89716bbad238142e3e9c8d57312aa896b154f1f9a9ae1bc72675f52091f"} err="failed to get container status \"8fc6f89716bbad238142e3e9c8d57312aa896b154f1f9a9ae1bc72675f52091f\": rpc error: code = NotFound desc = could not find container \"8fc6f89716bbad238142e3e9c8d57312aa896b154f1f9a9ae1bc72675f52091f\": container with ID starting with 8fc6f89716bbad238142e3e9c8d57312aa896b154f1f9a9ae1bc72675f52091f not found: ID does not exist" Nov 01 05:54:54.689363 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:54.689322 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-zwgwr"] Nov 01 05:54:54.696224 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:54.696197 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-zwgwr"] Nov 01 05:54:55.513107 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:54:55.513077 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5aa7cba-3da3-4424-a24f-0fcca53f4728" path="/var/lib/kubelet/pods/a5aa7cba-3da3-4424-a24f-0fcca53f4728/volumes" Nov 01 05:55:09.971662 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:55:09.971619 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 05:55:21.555439 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:55:21.555408 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 05:55:21.561465 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:55:21.561440 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 05:55:39.971763 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:55:39.971721 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 05:56:09.971736 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:56:09.971694 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 05:56:28.553819 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:28.553780 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl"] Nov 01 05:56:28.554198 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:28.554146 5040 cpu_manager.go:414] "containerMap: removing container" podUID="a5aa7cba-3da3-4424-a24f-0fcca53f4728" containerName="manager" Nov 01 05:56:28.554198 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:28.554156 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5aa7cba-3da3-4424-a24f-0fcca53f4728" containerName="manager" Nov 01 05:56:28.554273 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:28.554230 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5aa7cba-3da3-4424-a24f-0fcca53f4728" containerName="manager" Nov 01 05:56:28.557311 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:28.557292 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" Nov 01 05:56:28.599361 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:28.599325 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21f54836-d1ed-4f87-8f26-7c2d939b2590-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl\" (UID: \"21f54836-d1ed-4f87-8f26-7c2d939b2590\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" Nov 01 05:56:28.599561 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:28.599441 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21f54836-d1ed-4f87-8f26-7c2d939b2590-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl\" (UID: \"21f54836-d1ed-4f87-8f26-7c2d939b2590\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" Nov 01 05:56:28.599647 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:28.599574 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21f54836-d1ed-4f87-8f26-7c2d939b2590-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl\" (UID: \"21f54836-d1ed-4f87-8f26-7c2d939b2590\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" Nov 01 05:56:28.599647 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:28.599606 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21f54836-d1ed-4f87-8f26-7c2d939b2590-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl\" (UID: \"21f54836-d1ed-4f87-8f26-7c2d939b2590\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" Nov 01 05:56:28.599647 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:28.599632 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpdwp\" (UniqueName: \"kubernetes.io/projected/21f54836-d1ed-4f87-8f26-7c2d939b2590-kube-api-access-lpdwp\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl\" (UID: \"21f54836-d1ed-4f87-8f26-7c2d939b2590\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" Nov 01 05:56:28.599799 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:28.599707 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21f54836-d1ed-4f87-8f26-7c2d939b2590-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl\" (UID: \"21f54836-d1ed-4f87-8f26-7c2d939b2590\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" Nov 01 05:56:28.616667 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:28.616631 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl"] Nov 01 05:56:28.664056 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:28.664022 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 01 05:56:28.664300 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:28.664095 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 01 05:56:28.664300 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:28.664109 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 01 05:56:28.664300 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:28.664151 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4k6qc\"" Nov 01 05:56:28.700271 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:28.700232 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21f54836-d1ed-4f87-8f26-7c2d939b2590-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl\" (UID: \"21f54836-d1ed-4f87-8f26-7c2d939b2590\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" Nov 01 05:56:28.700271 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:28.700274 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21f54836-d1ed-4f87-8f26-7c2d939b2590-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl\" (UID: \"21f54836-d1ed-4f87-8f26-7c2d939b2590\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" Nov 01 05:56:28.700574 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:28.700294 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpdwp\" (UniqueName: \"kubernetes.io/projected/21f54836-d1ed-4f87-8f26-7c2d939b2590-kube-api-access-lpdwp\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl\" (UID: \"21f54836-d1ed-4f87-8f26-7c2d939b2590\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" Nov 01 05:56:28.700574 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:28.700454 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21f54836-d1ed-4f87-8f26-7c2d939b2590-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl\" (UID: \"21f54836-d1ed-4f87-8f26-7c2d939b2590\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" Nov 01 05:56:28.700719 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:28.700580 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21f54836-d1ed-4f87-8f26-7c2d939b2590-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl\" (UID: \"21f54836-d1ed-4f87-8f26-7c2d939b2590\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" Nov 01 05:56:28.700719 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:28.700631 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21f54836-d1ed-4f87-8f26-7c2d939b2590-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl\" (UID: \"21f54836-d1ed-4f87-8f26-7c2d939b2590\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" Nov 01 05:56:28.700827 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:28.700730 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21f54836-d1ed-4f87-8f26-7c2d939b2590-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl\" (UID: \"21f54836-d1ed-4f87-8f26-7c2d939b2590\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" Nov 01 05:56:28.700827 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:28.700806 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21f54836-d1ed-4f87-8f26-7c2d939b2590-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl\" (UID: \"21f54836-d1ed-4f87-8f26-7c2d939b2590\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" Nov 01 05:56:28.700990 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:28.700965 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21f54836-d1ed-4f87-8f26-7c2d939b2590-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl\" (UID: \"21f54836-d1ed-4f87-8f26-7c2d939b2590\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" Nov 01 05:56:28.703039 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:28.703012 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21f54836-d1ed-4f87-8f26-7c2d939b2590-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl\" (UID: \"21f54836-d1ed-4f87-8f26-7c2d939b2590\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" Nov 01 05:56:28.703466 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:28.703442 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21f54836-d1ed-4f87-8f26-7c2d939b2590-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl\" (UID: \"21f54836-d1ed-4f87-8f26-7c2d939b2590\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" Nov 01 05:56:28.762317 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:28.762277 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpdwp\" (UniqueName: \"kubernetes.io/projected/21f54836-d1ed-4f87-8f26-7c2d939b2590-kube-api-access-lpdwp\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl\" (UID: \"21f54836-d1ed-4f87-8f26-7c2d939b2590\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" Nov 01 05:56:28.868411 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:28.868321 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" Nov 01 05:56:29.244009 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:29.243977 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl"] Nov 01 05:56:29.246992 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:56:29.246965 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21f54836_d1ed_4f87_8f26_7c2d939b2590.slice/crio-c8671d93387905af56f22de2551485c0a21b847376c297f5292474833cae37b9 WatchSource:0}: Error finding container c8671d93387905af56f22de2551485c0a21b847376c297f5292474833cae37b9: Status 404 returned error can't find the container with id c8671d93387905af56f22de2551485c0a21b847376c297f5292474833cae37b9 Nov 01 05:56:29.249018 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:29.248993 5040 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 05:56:29.994989 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:29.994950 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" event={"ID":"21f54836-d1ed-4f87-8f26-7c2d939b2590","Type":"ContainerStarted","Data":"c8671d93387905af56f22de2551485c0a21b847376c297f5292474833cae37b9"} Nov 01 05:56:31.003657 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:31.003618 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" event={"ID":"21f54836-d1ed-4f87-8f26-7c2d939b2590","Type":"ContainerStarted","Data":"d9d7f6f7a47dd4885ccb3e3ff0f4fd4488dd3b7c0eedcd6cd00b2bc828907955"} Nov 01 05:56:31.125708 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:31.125654 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" podStartSLOduration=1.483414684 podStartE2EDuration="3.125639413s" totalImagesPullingTime="1.642224729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:56:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:56:31.12464433 +0000 UTC m=+669.779761499" watchObservedRunningTime="2026-11-01 05:56:31.125639413 +0000 UTC m=+669.780756579" Nov 01 05:56:31.868923 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:31.868888 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" Nov 01 05:56:38.869160 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:38.869121 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" Nov 01 05:56:38.881897 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:38.881872 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" Nov 01 05:56:39.056753 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:39.056723 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" Nov 01 05:56:39.973183 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:56:39.973141 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 05:56:59.122088 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:59.122058 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-558f494858-fkgrf"] Nov 01 05:56:59.125663 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:59.125646 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-558f494858-fkgrf" Nov 01 05:56:59.182039 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:59.182008 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-558f494858-fkgrf"] Nov 01 05:56:59.230472 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:59.230441 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wgl92\"" Nov 01 05:56:59.230472 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:59.230473 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 01 05:56:59.256572 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:59.256542 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ef1482fa-e45b-4784-939f-f6e9aa480c0e-tls-cert\") pod \"authorino-558f494858-fkgrf\" (UID: \"ef1482fa-e45b-4784-939f-f6e9aa480c0e\") " pod="kuadrant-system/authorino-558f494858-fkgrf" Nov 01 05:56:59.256719 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:59.256665 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spdmn\" (UniqueName: \"kubernetes.io/projected/ef1482fa-e45b-4784-939f-f6e9aa480c0e-kube-api-access-spdmn\") pod \"authorino-558f494858-fkgrf\" (UID: \"ef1482fa-e45b-4784-939f-f6e9aa480c0e\") " pod="kuadrant-system/authorino-558f494858-fkgrf" Nov 01 05:56:59.357228 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:59.357198 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spdmn\" (UniqueName: \"kubernetes.io/projected/ef1482fa-e45b-4784-939f-f6e9aa480c0e-kube-api-access-spdmn\") pod \"authorino-558f494858-fkgrf\" (UID: \"ef1482fa-e45b-4784-939f-f6e9aa480c0e\") " pod="kuadrant-system/authorino-558f494858-fkgrf" Nov 01 05:56:59.357385 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:59.357250 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ef1482fa-e45b-4784-939f-f6e9aa480c0e-tls-cert\") pod \"authorino-558f494858-fkgrf\" (UID: \"ef1482fa-e45b-4784-939f-f6e9aa480c0e\") " pod="kuadrant-system/authorino-558f494858-fkgrf" Nov 01 05:56:59.359588 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:59.359567 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ef1482fa-e45b-4784-939f-f6e9aa480c0e-tls-cert\") pod \"authorino-558f494858-fkgrf\" (UID: \"ef1482fa-e45b-4784-939f-f6e9aa480c0e\") " pod="kuadrant-system/authorino-558f494858-fkgrf" Nov 01 05:56:59.419359 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:59.419284 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spdmn\" (UniqueName: \"kubernetes.io/projected/ef1482fa-e45b-4784-939f-f6e9aa480c0e-kube-api-access-spdmn\") pod \"authorino-558f494858-fkgrf\" (UID: \"ef1482fa-e45b-4784-939f-f6e9aa480c0e\") " pod="kuadrant-system/authorino-558f494858-fkgrf" Nov 01 05:56:59.437075 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:59.437049 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-558f494858-fkgrf" Nov 01 05:56:59.817130 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:56:59.817100 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-558f494858-fkgrf"] Nov 01 05:56:59.818941 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:56:59.818899 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef1482fa_e45b_4784_939f_f6e9aa480c0e.slice/crio-bf7933802bfdb140269d7f7d971df682a0fc16acaab388840bfd0ea6d6e435e1 WatchSource:0}: Error finding container bf7933802bfdb140269d7f7d971df682a0fc16acaab388840bfd0ea6d6e435e1: Status 404 returned error can't find the container with id bf7933802bfdb140269d7f7d971df682a0fc16acaab388840bfd0ea6d6e435e1 Nov 01 05:57:00.150826 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:00.150739 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-558f494858-fkgrf" event={"ID":"ef1482fa-e45b-4784-939f-f6e9aa480c0e","Type":"ContainerStarted","Data":"bf7933802bfdb140269d7f7d971df682a0fc16acaab388840bfd0ea6d6e435e1"} Nov 01 05:57:02.165281 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:02.165236 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-558f494858-fkgrf" event={"ID":"ef1482fa-e45b-4784-939f-f6e9aa480c0e","Type":"ContainerStarted","Data":"a80deea9964bf5e02b76ca308d84f5c07f27808d7cef6a94dbcb5a5f01199bde"} Nov 01 05:57:02.282226 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:02.282166 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-558f494858-fkgrf" podStartSLOduration=1.784456475 podStartE2EDuration="3.282153424s" totalImagesPullingTime="1.497696949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:57:02.281220212 +0000 UTC m=+700.936337380" watchObservedRunningTime="2026-11-01 05:57:02.282153424 +0000 UTC m=+700.937270591" Nov 01 05:57:08.396379 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:08.396346 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h"] Nov 01 05:57:08.402071 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:08.402050 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" Nov 01 05:57:08.506745 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:08.506709 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h"] Nov 01 05:57:08.552945 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:08.552914 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba017ed-c86e-4442-a083-2a828ad624b1-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-tm67h\" (UID: \"bba017ed-c86e-4442-a083-2a828ad624b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" Nov 01 05:57:08.553148 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:08.552958 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba017ed-c86e-4442-a083-2a828ad624b1-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-tm67h\" (UID: \"bba017ed-c86e-4442-a083-2a828ad624b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" Nov 01 05:57:08.553148 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:08.553065 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba017ed-c86e-4442-a083-2a828ad624b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-tm67h\" (UID: \"bba017ed-c86e-4442-a083-2a828ad624b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" Nov 01 05:57:08.553148 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:08.553090 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba017ed-c86e-4442-a083-2a828ad624b1-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-tm67h\" (UID: \"bba017ed-c86e-4442-a083-2a828ad624b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" Nov 01 05:57:08.553148 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:08.553114 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns672\" (UniqueName: \"kubernetes.io/projected/bba017ed-c86e-4442-a083-2a828ad624b1-kube-api-access-ns672\") pod \"maas-default-gateway-openshift-default-855f9574cf-tm67h\" (UID: \"bba017ed-c86e-4442-a083-2a828ad624b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" Nov 01 05:57:08.553398 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:08.553185 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba017ed-c86e-4442-a083-2a828ad624b1-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-tm67h\" (UID: \"bba017ed-c86e-4442-a083-2a828ad624b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" Nov 01 05:57:08.553398 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:08.553235 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba017ed-c86e-4442-a083-2a828ad624b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-tm67h\" (UID: \"bba017ed-c86e-4442-a083-2a828ad624b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" Nov 01 05:57:08.553398 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:08.553350 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba017ed-c86e-4442-a083-2a828ad624b1-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-tm67h\" (UID: \"bba017ed-c86e-4442-a083-2a828ad624b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" Nov 01 05:57:08.553398 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:08.553385 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba017ed-c86e-4442-a083-2a828ad624b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-tm67h\" (UID: \"bba017ed-c86e-4442-a083-2a828ad624b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" Nov 01 05:57:08.654292 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:08.654201 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba017ed-c86e-4442-a083-2a828ad624b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-tm67h\" (UID: \"bba017ed-c86e-4442-a083-2a828ad624b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" Nov 01 05:57:08.654292 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:08.654237 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba017ed-c86e-4442-a083-2a828ad624b1-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-tm67h\" (UID: \"bba017ed-c86e-4442-a083-2a828ad624b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" Nov 01 05:57:08.654588 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:08.654342 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns672\" (UniqueName: \"kubernetes.io/projected/bba017ed-c86e-4442-a083-2a828ad624b1-kube-api-access-ns672\") pod \"maas-default-gateway-openshift-default-855f9574cf-tm67h\" (UID: \"bba017ed-c86e-4442-a083-2a828ad624b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" Nov 01 05:57:08.654588 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:08.654379 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba017ed-c86e-4442-a083-2a828ad624b1-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-tm67h\" (UID: \"bba017ed-c86e-4442-a083-2a828ad624b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" Nov 01 05:57:08.654588 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:08.654403 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba017ed-c86e-4442-a083-2a828ad624b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-tm67h\" (UID: \"bba017ed-c86e-4442-a083-2a828ad624b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" Nov 01 05:57:08.654588 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:08.654467 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba017ed-c86e-4442-a083-2a828ad624b1-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-tm67h\" (UID: \"bba017ed-c86e-4442-a083-2a828ad624b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" Nov 01 05:57:08.654588 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:08.654528 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba017ed-c86e-4442-a083-2a828ad624b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-tm67h\" (UID: \"bba017ed-c86e-4442-a083-2a828ad624b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" Nov 01 05:57:08.654588 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:08.654576 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba017ed-c86e-4442-a083-2a828ad624b1-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-tm67h\" (UID: \"bba017ed-c86e-4442-a083-2a828ad624b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" Nov 01 05:57:08.654912 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:08.654607 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba017ed-c86e-4442-a083-2a828ad624b1-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-tm67h\" (UID: \"bba017ed-c86e-4442-a083-2a828ad624b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" Nov 01 05:57:08.654912 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:08.654823 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba017ed-c86e-4442-a083-2a828ad624b1-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-tm67h\" (UID: \"bba017ed-c86e-4442-a083-2a828ad624b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" Nov 01 05:57:08.654912 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:08.654860 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba017ed-c86e-4442-a083-2a828ad624b1-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-tm67h\" (UID: \"bba017ed-c86e-4442-a083-2a828ad624b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" Nov 01 05:57:08.655032 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:08.654907 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba017ed-c86e-4442-a083-2a828ad624b1-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-tm67h\" (UID: \"bba017ed-c86e-4442-a083-2a828ad624b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" Nov 01 05:57:08.655032 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:08.654939 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba017ed-c86e-4442-a083-2a828ad624b1-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-tm67h\" (UID: \"bba017ed-c86e-4442-a083-2a828ad624b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" Nov 01 05:57:08.655541 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:08.655517 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba017ed-c86e-4442-a083-2a828ad624b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-tm67h\" (UID: \"bba017ed-c86e-4442-a083-2a828ad624b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" Nov 01 05:57:08.656818 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:08.656799 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba017ed-c86e-4442-a083-2a828ad624b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-tm67h\" (UID: \"bba017ed-c86e-4442-a083-2a828ad624b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" Nov 01 05:57:08.656869 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:08.656859 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba017ed-c86e-4442-a083-2a828ad624b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-tm67h\" (UID: \"bba017ed-c86e-4442-a083-2a828ad624b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" Nov 01 05:57:08.716099 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:08.716063 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba017ed-c86e-4442-a083-2a828ad624b1-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-tm67h\" (UID: \"bba017ed-c86e-4442-a083-2a828ad624b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" Nov 01 05:57:08.716244 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:08.716157 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns672\" (UniqueName: \"kubernetes.io/projected/bba017ed-c86e-4442-a083-2a828ad624b1-kube-api-access-ns672\") pod \"maas-default-gateway-openshift-default-855f9574cf-tm67h\" (UID: \"bba017ed-c86e-4442-a083-2a828ad624b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" Nov 01 05:57:09.015179 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:09.015143 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" Nov 01 05:57:09.440189 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:09.440162 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h"] Nov 01 05:57:09.441490 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:57:09.441446 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba017ed_c86e_4442_a083_2a828ad624b1.slice/crio-197233d60215c8b49367f04a5efe12aaffc91ace195d965d0e66fddfec52ff54 WatchSource:0}: Error finding container 197233d60215c8b49367f04a5efe12aaffc91ace195d965d0e66fddfec52ff54: Status 404 returned error can't find the container with id 197233d60215c8b49367f04a5efe12aaffc91ace195d965d0e66fddfec52ff54 Nov 01 05:57:09.443551 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:09.443518 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 05:57:09.443623 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:09.443587 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 05:57:09.443659 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:09.443617 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 05:57:09.971491 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:57:09.971442 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 05:57:10.223103 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:10.223002 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" event={"ID":"bba017ed-c86e-4442-a083-2a828ad624b1","Type":"ContainerStarted","Data":"2716189ca52914194cf6269d27c0fd8bb78d3d60b8c9d57a9fcc0fff13eee741"} Nov 01 05:57:10.223103 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:10.223095 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" event={"ID":"bba017ed-c86e-4442-a083-2a828ad624b1","Type":"ContainerStarted","Data":"197233d60215c8b49367f04a5efe12aaffc91ace195d965d0e66fddfec52ff54"} Nov 01 05:57:10.461613 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:10.461555 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" podStartSLOduration=2.46154051 podStartE2EDuration="2.46154051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:57:10.409713939 +0000 UTC m=+709.064831106" watchObservedRunningTime="2026-11-01 05:57:10.46154051 +0000 UTC m=+709.116657739" Nov 01 05:57:11.017168 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:11.017136 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" Nov 01 05:57:11.017601 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:11.017572 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tm67h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 01 05:57:11.017793 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:11.017768 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" podUID="bba017ed-c86e-4442-a083-2a828ad624b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 01 05:57:12.016633 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:12.016599 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tm67h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 01 05:57:12.017333 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:12.017275 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" podUID="bba017ed-c86e-4442-a083-2a828ad624b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 01 05:57:13.020354 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:13.020298 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tm67h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 01 05:57:13.020875 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:13.020428 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" podUID="bba017ed-c86e-4442-a083-2a828ad624b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 01 05:57:14.016503 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:14.015885 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tm67h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 01 05:57:14.016503 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:14.015948 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" podUID="bba017ed-c86e-4442-a083-2a828ad624b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 01 05:57:15.018505 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:15.018267 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tm67h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 01 05:57:15.018505 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:15.018327 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" podUID="bba017ed-c86e-4442-a083-2a828ad624b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 01 05:57:16.015885 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:16.015844 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tm67h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 01 05:57:16.016067 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:16.015905 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" podUID="bba017ed-c86e-4442-a083-2a828ad624b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 01 05:57:17.523391 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:17.523363 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" Nov 01 05:57:17.525076 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:17.525057 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" Nov 01 05:57:17.525789 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:57:17.525763 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" Nov 01 05:57:39.971763 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:57:39.971723 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 05:58:09.972876 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:58:09.972831 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 05:58:39.973499 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:58:39.973438 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 05:58:54.027217 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:54.027175 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg"] Nov 01 05:58:54.030770 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:54.030747 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" Nov 01 05:58:54.057392 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:54.057366 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-k99rg\" (UID: \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" Nov 01 05:58:54.057538 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:54.057396 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-k99rg\" (UID: \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" Nov 01 05:58:54.057538 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:54.057424 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-k99rg\" (UID: \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" Nov 01 05:58:54.057634 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:54.057576 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-k99rg\" (UID: \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" Nov 01 05:58:54.057681 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:54.057646 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rblg\" (UniqueName: \"kubernetes.io/projected/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-kube-api-access-5rblg\") pod \"maas-default-gateway-openshift-default-855f9574cf-k99rg\" (UID: \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" Nov 01 05:58:54.057771 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:54.057754 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-k99rg\" (UID: \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" Nov 01 05:58:54.057812 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:54.057788 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-k99rg\" (UID: \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" Nov 01 05:58:54.057854 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:54.057825 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-k99rg\" (UID: \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" Nov 01 05:58:54.057894 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:54.057879 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-k99rg\" (UID: \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" Nov 01 05:58:54.129355 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:54.129321 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg"] Nov 01 05:58:54.158352 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:54.158320 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rblg\" (UniqueName: \"kubernetes.io/projected/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-kube-api-access-5rblg\") pod \"maas-default-gateway-openshift-default-855f9574cf-k99rg\" (UID: \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" Nov 01 05:58:54.158533 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:54.158380 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-k99rg\" (UID: \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" Nov 01 05:58:54.158533 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:54.158407 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-k99rg\" (UID: \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" Nov 01 05:58:54.158533 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:54.158434 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-k99rg\" (UID: \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" Nov 01 05:58:54.158533 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:54.158517 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-k99rg\" (UID: \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" Nov 01 05:58:54.158762 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:54.158547 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-k99rg\" (UID: \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" Nov 01 05:58:54.158762 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:54.158570 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-k99rg\" (UID: \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" Nov 01 05:58:54.158762 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:54.158602 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-k99rg\" (UID: \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" Nov 01 05:58:54.158762 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:54.158642 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-k99rg\" (UID: \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" Nov 01 05:58:54.159429 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:54.159408 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-k99rg\" (UID: \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" Nov 01 05:58:54.160830 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:54.160807 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-k99rg\" (UID: \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" Nov 01 05:58:54.161229 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:54.161208 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-k99rg\" (UID: \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" Nov 01 05:58:54.162468 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:54.161556 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-k99rg\" (UID: \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" Nov 01 05:58:54.162468 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:54.161686 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-k99rg\" (UID: \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" Nov 01 05:58:54.162944 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:54.162922 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-k99rg\" (UID: \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" Nov 01 05:58:54.163914 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:54.163891 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-k99rg\" (UID: \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" Nov 01 05:58:54.220696 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:54.220664 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-k99rg\" (UID: \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" Nov 01 05:58:54.220829 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:54.220762 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rblg\" (UniqueName: \"kubernetes.io/projected/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-kube-api-access-5rblg\") pod \"maas-default-gateway-openshift-default-855f9574cf-k99rg\" (UID: \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" Nov 01 05:58:54.343768 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:54.343685 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" Nov 01 05:58:54.771987 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:54.771954 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg"] Nov 01 05:58:54.773863 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:58:54.773833 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d2bd89_b64a_4f9e_ba55_d9cb9caf1907.slice/crio-ee880f82dc7fed54f77921561d5f23bc8d2994311a3c1749158313c9dd7b15a9 WatchSource:0}: Error finding container ee880f82dc7fed54f77921561d5f23bc8d2994311a3c1749158313c9dd7b15a9: Status 404 returned error can't find the container with id ee880f82dc7fed54f77921561d5f23bc8d2994311a3c1749158313c9dd7b15a9 Nov 01 05:58:54.776036 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:54.776006 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 05:58:54.776116 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:54.776068 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 05:58:54.776116 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:54.776102 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 05:58:55.782775 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:55.782733 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" event={"ID":"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907","Type":"ContainerStarted","Data":"a83a4164778bda232058de40b85b77529ada83cb1cb1bba8ca179f21b9c16a97"} Nov 01 05:58:55.783163 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:55.782791 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" event={"ID":"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907","Type":"ContainerStarted","Data":"ee880f82dc7fed54f77921561d5f23bc8d2994311a3c1749158313c9dd7b15a9"} Nov 01 05:58:56.018503 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:56.017824 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" podStartSLOduration=3.017802086 podStartE2EDuration="3.017802086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:58:55.955080776 +0000 UTC m=+814.610197944" watchObservedRunningTime="2026-11-01 05:58:56.017802086 +0000 UTC m=+814.672919255" Nov 01 05:58:56.344694 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:56.344662 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" Nov 01 05:58:56.346193 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:56.346166 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-k99rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 01 05:58:56.346314 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:56.346212 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" podUID="67d2bd89-b64a-4f9e-ba55-d9cb9caf1907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 01 05:58:57.344766 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:57.344726 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-k99rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 01 05:58:57.345179 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:57.344788 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" podUID="67d2bd89-b64a-4f9e-ba55-d9cb9caf1907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 01 05:58:58.344785 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:58.344744 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-k99rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 01 05:58:58.345278 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:58.344806 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" podUID="67d2bd89-b64a-4f9e-ba55-d9cb9caf1907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 01 05:58:59.344839 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:59.344802 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-k99rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 01 05:58:59.345233 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:58:59.344855 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" podUID="67d2bd89-b64a-4f9e-ba55-d9cb9caf1907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 01 05:59:00.344610 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:00.344575 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-k99rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 01 05:59:00.344808 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:00.344636 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" podUID="67d2bd89-b64a-4f9e-ba55-d9cb9caf1907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 01 05:59:01.344980 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:01.344947 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-k99rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 01 05:59:01.345358 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:01.345004 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" podUID="67d2bd89-b64a-4f9e-ba55-d9cb9caf1907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 01 05:59:02.344595 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:02.344559 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-k99rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 01 05:59:02.344774 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:02.344612 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" podUID="67d2bd89-b64a-4f9e-ba55-d9cb9caf1907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 01 05:59:03.348992 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:03.348955 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" Nov 01 05:59:03.349366 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:03.349334 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" Nov 01 05:59:03.350088 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:03.350069 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" Nov 01 05:59:09.042684 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:09.042646 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf"] Nov 01 05:59:09.046445 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:09.046423 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" Nov 01 05:59:09.111703 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:09.111667 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aeeb98d-3738-4319-8531-dba6f0705696-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-d5hgf\" (UID: \"0aeeb98d-3738-4319-8531-dba6f0705696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" Nov 01 05:59:09.111899 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:09.111730 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aeeb98d-3738-4319-8531-dba6f0705696-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-d5hgf\" (UID: \"0aeeb98d-3738-4319-8531-dba6f0705696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" Nov 01 05:59:09.111899 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:09.111757 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aeeb98d-3738-4319-8531-dba6f0705696-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-d5hgf\" (UID: \"0aeeb98d-3738-4319-8531-dba6f0705696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" Nov 01 05:59:09.111899 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:09.111781 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7jgd\" (UniqueName: \"kubernetes.io/projected/0aeeb98d-3738-4319-8531-dba6f0705696-kube-api-access-l7jgd\") pod \"maas-default-gateway-openshift-default-855f9574cf-d5hgf\" (UID: \"0aeeb98d-3738-4319-8531-dba6f0705696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" Nov 01 05:59:09.112047 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:09.111923 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aeeb98d-3738-4319-8531-dba6f0705696-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-d5hgf\" (UID: \"0aeeb98d-3738-4319-8531-dba6f0705696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" Nov 01 05:59:09.112047 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:09.111972 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aeeb98d-3738-4319-8531-dba6f0705696-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-d5hgf\" (UID: \"0aeeb98d-3738-4319-8531-dba6f0705696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" Nov 01 05:59:09.112047 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:09.111990 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aeeb98d-3738-4319-8531-dba6f0705696-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-d5hgf\" (UID: \"0aeeb98d-3738-4319-8531-dba6f0705696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" Nov 01 05:59:09.112199 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:09.112102 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aeeb98d-3738-4319-8531-dba6f0705696-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-d5hgf\" (UID: \"0aeeb98d-3738-4319-8531-dba6f0705696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" Nov 01 05:59:09.112199 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:09.112150 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aeeb98d-3738-4319-8531-dba6f0705696-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-d5hgf\" (UID: \"0aeeb98d-3738-4319-8531-dba6f0705696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" Nov 01 05:59:09.212735 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:09.212699 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aeeb98d-3738-4319-8531-dba6f0705696-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-d5hgf\" (UID: \"0aeeb98d-3738-4319-8531-dba6f0705696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" Nov 01 05:59:09.212889 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:09.212749 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aeeb98d-3738-4319-8531-dba6f0705696-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-d5hgf\" (UID: \"0aeeb98d-3738-4319-8531-dba6f0705696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" Nov 01 05:59:09.212889 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:09.212792 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aeeb98d-3738-4319-8531-dba6f0705696-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-d5hgf\" (UID: \"0aeeb98d-3738-4319-8531-dba6f0705696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" Nov 01 05:59:09.212981 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:09.212924 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7jgd\" (UniqueName: \"kubernetes.io/projected/0aeeb98d-3738-4319-8531-dba6f0705696-kube-api-access-l7jgd\") pod \"maas-default-gateway-openshift-default-855f9574cf-d5hgf\" (UID: \"0aeeb98d-3738-4319-8531-dba6f0705696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" Nov 01 05:59:09.213042 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:09.213022 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aeeb98d-3738-4319-8531-dba6f0705696-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-d5hgf\" (UID: \"0aeeb98d-3738-4319-8531-dba6f0705696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" Nov 01 05:59:09.213092 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:09.213076 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aeeb98d-3738-4319-8531-dba6f0705696-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-d5hgf\" (UID: \"0aeeb98d-3738-4319-8531-dba6f0705696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" Nov 01 05:59:09.213143 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:09.213105 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aeeb98d-3738-4319-8531-dba6f0705696-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-d5hgf\" (UID: \"0aeeb98d-3738-4319-8531-dba6f0705696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" Nov 01 05:59:09.213208 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:09.213182 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aeeb98d-3738-4319-8531-dba6f0705696-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-d5hgf\" (UID: \"0aeeb98d-3738-4319-8531-dba6f0705696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" Nov 01 05:59:09.213267 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:09.213214 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aeeb98d-3738-4319-8531-dba6f0705696-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-d5hgf\" (UID: \"0aeeb98d-3738-4319-8531-dba6f0705696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" Nov 01 05:59:09.213267 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:09.213229 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aeeb98d-3738-4319-8531-dba6f0705696-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-d5hgf\" (UID: \"0aeeb98d-3738-4319-8531-dba6f0705696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" Nov 01 05:59:09.213361 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:09.213276 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aeeb98d-3738-4319-8531-dba6f0705696-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-d5hgf\" (UID: \"0aeeb98d-3738-4319-8531-dba6f0705696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" Nov 01 05:59:09.213463 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:09.213442 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aeeb98d-3738-4319-8531-dba6f0705696-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-d5hgf\" (UID: \"0aeeb98d-3738-4319-8531-dba6f0705696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" Nov 01 05:59:09.213557 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:09.213540 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aeeb98d-3738-4319-8531-dba6f0705696-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-d5hgf\" (UID: \"0aeeb98d-3738-4319-8531-dba6f0705696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" Nov 01 05:59:09.213660 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:09.213642 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aeeb98d-3738-4319-8531-dba6f0705696-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-d5hgf\" (UID: \"0aeeb98d-3738-4319-8531-dba6f0705696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" Nov 01 05:59:09.215407 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:09.215385 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aeeb98d-3738-4319-8531-dba6f0705696-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-d5hgf\" (UID: \"0aeeb98d-3738-4319-8531-dba6f0705696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" Nov 01 05:59:09.215756 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:09.215730 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aeeb98d-3738-4319-8531-dba6f0705696-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-d5hgf\" (UID: \"0aeeb98d-3738-4319-8531-dba6f0705696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" Nov 01 05:59:09.247340 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:09.247309 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf"] Nov 01 05:59:09.274553 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:09.274519 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aeeb98d-3738-4319-8531-dba6f0705696-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-d5hgf\" (UID: \"0aeeb98d-3738-4319-8531-dba6f0705696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" Nov 01 05:59:09.274699 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:09.274588 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7jgd\" (UniqueName: \"kubernetes.io/projected/0aeeb98d-3738-4319-8531-dba6f0705696-kube-api-access-l7jgd\") pod \"maas-default-gateway-openshift-default-855f9574cf-d5hgf\" (UID: \"0aeeb98d-3738-4319-8531-dba6f0705696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" Nov 01 05:59:09.360954 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:09.360876 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" Nov 01 05:59:09.788551 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:09.788516 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf"] Nov 01 05:59:09.790258 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:59:09.790224 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aeeb98d_3738_4319_8531_dba6f0705696.slice/crio-e3cf79fd28a63e41c46508eb27de97b12f22891f97ee622572f4819332b7770f WatchSource:0}: Error finding container e3cf79fd28a63e41c46508eb27de97b12f22891f97ee622572f4819332b7770f: Status 404 returned error can't find the container with id e3cf79fd28a63e41c46508eb27de97b12f22891f97ee622572f4819332b7770f Nov 01 05:59:09.792602 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:09.792573 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 05:59:09.792671 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:09.792639 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 05:59:09.792707 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:09.792674 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 05:59:09.882258 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:09.882222 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" event={"ID":"0aeeb98d-3738-4319-8531-dba6f0705696","Type":"ContainerStarted","Data":"9799a124214b067052467c5b105639c3ca2461227b57c8eeba925d26f3566501"} Nov 01 05:59:09.882394 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:09.882279 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" event={"ID":"0aeeb98d-3738-4319-8531-dba6f0705696","Type":"ContainerStarted","Data":"e3cf79fd28a63e41c46508eb27de97b12f22891f97ee622572f4819332b7770f"} Nov 01 05:59:09.973268 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:59:09.973230 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 05:59:10.006326 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:10.006276 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" podStartSLOduration=2.006259924 podStartE2EDuration="2.006259924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:59:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:59:10.003127243 +0000 UTC m=+828.658244410" watchObservedRunningTime="2026-11-01 05:59:10.006259924 +0000 UTC m=+828.661377091" Nov 01 05:59:10.361392 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:10.361359 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" Nov 01 05:59:10.363192 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:10.363168 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-d5hgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 01 05:59:10.363309 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:10.363223 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" podUID="0aeeb98d-3738-4319-8531-dba6f0705696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 01 05:59:11.361408 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:11.361375 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-d5hgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 01 05:59:11.361901 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:11.361433 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" podUID="0aeeb98d-3738-4319-8531-dba6f0705696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 01 05:59:12.361424 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:12.361394 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-d5hgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 01 05:59:12.361831 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:12.361441 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" podUID="0aeeb98d-3738-4319-8531-dba6f0705696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 01 05:59:13.361973 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:13.361941 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-d5hgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 01 05:59:13.362366 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:13.361991 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" podUID="0aeeb98d-3738-4319-8531-dba6f0705696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 01 05:59:14.361846 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:14.361805 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-d5hgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 01 05:59:14.362006 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:14.361861 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" podUID="0aeeb98d-3738-4319-8531-dba6f0705696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 01 05:59:15.361531 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:15.361464 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-d5hgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 01 05:59:15.361691 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:15.361554 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" podUID="0aeeb98d-3738-4319-8531-dba6f0705696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 01 05:59:16.361673 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:16.361632 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-d5hgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 01 05:59:16.362172 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:16.361683 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" podUID="0aeeb98d-3738-4319-8531-dba6f0705696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 01 05:59:18.135255 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:18.135203 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" Nov 01 05:59:18.136015 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:18.135984 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" Nov 01 05:59:18.136947 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:18.136927 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" Nov 01 05:59:24.071256 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:24.071221 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4"] Nov 01 05:59:24.074987 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:24.074970 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" Nov 01 05:59:24.133331 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:24.133298 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4"] Nov 01 05:59:24.154751 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:24.154722 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-tls-certs\") pod \"simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4\" (UID: \"1878b6fb-f8fe-4f4f-8f4e-056c66fc206a\") " pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" Nov 01 05:59:24.154919 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:24.154758 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-dshm\") pod \"simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4\" (UID: \"1878b6fb-f8fe-4f4f-8f4e-056c66fc206a\") " pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" Nov 01 05:59:24.154919 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:24.154778 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-model-cache\") pod \"simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4\" (UID: \"1878b6fb-f8fe-4f4f-8f4e-056c66fc206a\") " pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" Nov 01 05:59:24.154919 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:24.154905 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-home\") pod \"simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4\" (UID: \"1878b6fb-f8fe-4f4f-8f4e-056c66fc206a\") " pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" Nov 01 05:59:24.155045 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:24.154927 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95jjb\" (UniqueName: \"kubernetes.io/projected/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-kube-api-access-95jjb\") pod \"simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4\" (UID: \"1878b6fb-f8fe-4f4f-8f4e-056c66fc206a\") " pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" Nov 01 05:59:24.155045 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:24.154958 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-tmp-dir\") pod \"simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4\" (UID: \"1878b6fb-f8fe-4f4f-8f4e-056c66fc206a\") " pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" Nov 01 05:59:24.181199 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:24.181167 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-14a6e4\"/\"simulator-w2-14a6e4-kserve-self-signed-certs\"" Nov 01 05:59:24.181199 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:24.181190 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-14a6e4\"/\"default-dockercfg-qptrp\"" Nov 01 05:59:24.181388 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:24.181175 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-14a6e4\"/\"openshift-service-ca.crt\"" Nov 01 05:59:24.181388 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:24.181166 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-14a6e4\"/\"kube-root-ca.crt\"" Nov 01 05:59:24.255911 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:24.255878 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-tls-certs\") pod \"simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4\" (UID: \"1878b6fb-f8fe-4f4f-8f4e-056c66fc206a\") " pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" Nov 01 05:59:24.255911 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:24.255917 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-dshm\") pod \"simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4\" (UID: \"1878b6fb-f8fe-4f4f-8f4e-056c66fc206a\") " pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" Nov 01 05:59:24.256133 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:24.255936 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-model-cache\") pod \"simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4\" (UID: \"1878b6fb-f8fe-4f4f-8f4e-056c66fc206a\") " pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" Nov 01 05:59:24.256133 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:24.255984 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-home\") pod \"simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4\" (UID: \"1878b6fb-f8fe-4f4f-8f4e-056c66fc206a\") " pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" Nov 01 05:59:24.256133 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:24.256017 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95jjb\" (UniqueName: \"kubernetes.io/projected/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-kube-api-access-95jjb\") pod \"simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4\" (UID: \"1878b6fb-f8fe-4f4f-8f4e-056c66fc206a\") " pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" Nov 01 05:59:24.256133 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:24.256065 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-tmp-dir\") pod \"simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4\" (UID: \"1878b6fb-f8fe-4f4f-8f4e-056c66fc206a\") " pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" Nov 01 05:59:24.256503 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:24.256457 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-home\") pod \"simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4\" (UID: \"1878b6fb-f8fe-4f4f-8f4e-056c66fc206a\") " pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" Nov 01 05:59:24.256658 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:24.256466 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-tmp-dir\") pod \"simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4\" (UID: \"1878b6fb-f8fe-4f4f-8f4e-056c66fc206a\") " pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" Nov 01 05:59:24.256658 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:24.256583 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-model-cache\") pod \"simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4\" (UID: \"1878b6fb-f8fe-4f4f-8f4e-056c66fc206a\") " pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" Nov 01 05:59:24.258292 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:24.258270 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-dshm\") pod \"simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4\" (UID: \"1878b6fb-f8fe-4f4f-8f4e-056c66fc206a\") " pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" Nov 01 05:59:24.258633 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:24.258613 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-tls-certs\") pod \"simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4\" (UID: \"1878b6fb-f8fe-4f4f-8f4e-056c66fc206a\") " pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" Nov 01 05:59:24.317930 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:24.317898 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95jjb\" (UniqueName: \"kubernetes.io/projected/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-kube-api-access-95jjb\") pod \"simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4\" (UID: \"1878b6fb-f8fe-4f4f-8f4e-056c66fc206a\") " pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" Nov 01 05:59:24.386743 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:24.386659 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" Nov 01 05:59:24.771648 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:24.771619 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4"] Nov 01 05:59:24.773582 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:59:24.773551 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1878b6fb_f8fe_4f4f_8f4e_056c66fc206a.slice/crio-4a84add56d4c1d11c1360f4b96bbaab655138f1cc064e672cf5b54eaf8fd0ccd WatchSource:0}: Error finding container 4a84add56d4c1d11c1360f4b96bbaab655138f1cc064e672cf5b54eaf8fd0ccd: Status 404 returned error can't find the container with id 4a84add56d4c1d11c1360f4b96bbaab655138f1cc064e672cf5b54eaf8fd0ccd Nov 01 05:59:24.997381 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:24.997343 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" event={"ID":"1878b6fb-f8fe-4f4f-8f4e-056c66fc206a","Type":"ContainerStarted","Data":"d0dae898e24f105768cd27fbef7c830ee1cda3094e8cfd3b99fae27478517cf0"} Nov 01 05:59:24.997681 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:24.997391 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" event={"ID":"1878b6fb-f8fe-4f4f-8f4e-056c66fc206a","Type":"ContainerStarted","Data":"4a84add56d4c1d11c1360f4b96bbaab655138f1cc064e672cf5b54eaf8fd0ccd"} Nov 01 05:59:25.117721 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:25.117601 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" podStartSLOduration=1.117587693 podStartE2EDuration="1.117587693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:59:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:59:25.115630038 +0000 UTC m=+843.770747206" watchObservedRunningTime="2026-11-01 05:59:25.117587693 +0000 UTC m=+843.772704879" Nov 01 05:59:25.387453 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:25.387376 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" Nov 01 05:59:34.387548 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:34.387433 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" Nov 01 05:59:34.400135 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:34.400106 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" Nov 01 05:59:35.064438 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:35.064407 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" Nov 01 05:59:36.709863 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:36.709829 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp"] Nov 01 05:59:36.713495 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:36.713457 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" Nov 01 05:59:36.766977 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:36.766945 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-9c8fc5-openshift-default-dockercfg-6mn84\"" Nov 01 05:59:36.788252 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:36.788225 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-workload-certs\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp\" (UID: \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" Nov 01 05:59:36.788404 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:36.788259 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-istio-token\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp\" (UID: \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" Nov 01 05:59:36.788505 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:36.788418 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-istiod-ca-cert\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp\" (UID: \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" Nov 01 05:59:36.788505 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:36.788457 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-istio-data\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp\" (UID: \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" Nov 01 05:59:36.788594 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:36.788569 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-credential-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp\" (UID: \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" Nov 01 05:59:36.788640 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:36.788605 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxvx6\" (UniqueName: \"kubernetes.io/projected/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-kube-api-access-kxvx6\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp\" (UID: \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" Nov 01 05:59:36.788640 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:36.788622 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-istio-envoy\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp\" (UID: \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" Nov 01 05:59:36.788743 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:36.788642 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-workload-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp\" (UID: \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" Nov 01 05:59:36.788743 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:36.788689 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-istio-podinfo\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp\" (UID: \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" Nov 01 05:59:36.821082 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:36.821053 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp"] Nov 01 05:59:36.889955 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:36.889914 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-istiod-ca-cert\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp\" (UID: \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" Nov 01 05:59:36.890154 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:36.889964 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-istio-data\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp\" (UID: \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" Nov 01 05:59:36.890154 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:36.890010 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-credential-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp\" (UID: \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" Nov 01 05:59:36.890154 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:36.890046 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxvx6\" (UniqueName: \"kubernetes.io/projected/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-kube-api-access-kxvx6\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp\" (UID: \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" Nov 01 05:59:36.890154 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:36.890073 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-istio-envoy\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp\" (UID: \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" Nov 01 05:59:36.890154 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:36.890106 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-workload-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp\" (UID: \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" Nov 01 05:59:36.890154 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:36.890129 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-istio-podinfo\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp\" (UID: \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" Nov 01 05:59:36.890493 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:36.890174 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-workload-certs\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp\" (UID: \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" Nov 01 05:59:36.890493 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:36.890204 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-istio-token\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp\" (UID: \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" Nov 01 05:59:36.890613 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:36.890543 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-workload-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp\" (UID: \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" Nov 01 05:59:36.890826 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:36.890801 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-istiod-ca-cert\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp\" (UID: \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" Nov 01 05:59:36.890938 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:36.890917 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-credential-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp\" (UID: \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" Nov 01 05:59:36.891008 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:36.890983 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-workload-certs\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp\" (UID: \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" Nov 01 05:59:36.891062 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:36.891045 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-istio-data\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp\" (UID: \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" Nov 01 05:59:36.892607 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:36.892583 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-istio-envoy\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp\" (UID: \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" Nov 01 05:59:36.892964 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:36.892947 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-istio-podinfo\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp\" (UID: \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" Nov 01 05:59:36.952166 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:36.952128 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-istio-token\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp\" (UID: \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" Nov 01 05:59:36.952322 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:36.952201 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxvx6\" (UniqueName: \"kubernetes.io/projected/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-kube-api-access-kxvx6\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp\" (UID: \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" Nov 01 05:59:37.027224 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:37.027186 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" Nov 01 05:59:37.456118 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:37.456090 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp"] Nov 01 05:59:37.457387 ip-10-0-143-195 kubenswrapper[5040]: W1101 05:59:37.457361 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf77172_40a2_4853_9e6a_02a0ed2cbc4b.slice/crio-55100d39f80869bad1168fdbfbc2e1ed0136ea532cd2814a2ebe4a09577c780c WatchSource:0}: Error finding container 55100d39f80869bad1168fdbfbc2e1ed0136ea532cd2814a2ebe4a09577c780c: Status 404 returned error can't find the container with id 55100d39f80869bad1168fdbfbc2e1ed0136ea532cd2814a2ebe4a09577c780c Nov 01 05:59:37.459629 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:37.459595 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 05:59:37.459698 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:37.459666 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 05:59:37.459698 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:37.459693 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 05:59:38.071415 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:38.071379 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" event={"ID":"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b","Type":"ContainerStarted","Data":"e0467cec0d126bfccecd68df4cfbdcc9825d3d2627a83cdeba6c1a4955a9f0fb"} Nov 01 05:59:38.071811 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:38.071443 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" event={"ID":"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b","Type":"ContainerStarted","Data":"55100d39f80869bad1168fdbfbc2e1ed0136ea532cd2814a2ebe4a09577c780c"} Nov 01 05:59:38.193041 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:38.192989 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" podStartSLOduration=2.192975883 podStartE2EDuration="2.192975883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:59:38.192840701 +0000 UTC m=+856.847957869" watchObservedRunningTime="2026-11-01 05:59:38.192975883 +0000 UTC m=+856.848093050" Nov 01 05:59:39.027942 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:39.027906 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" Nov 01 05:59:39.029577 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:39.029551 5040 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 01 05:59:39.029702 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:39.029608 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" podUID="8bf77172-40a2-4853-9e6a-02a0ed2cbc4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 01 05:59:39.972069 ip-10-0-143-195 kubenswrapper[5040]: E1101 05:59:39.972029 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 05:59:40.028035 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:40.027997 5040 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 01 05:59:40.028221 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:40.028048 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" podUID="8bf77172-40a2-4853-9e6a-02a0ed2cbc4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 01 05:59:41.027704 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:41.027670 5040 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 01 05:59:41.028079 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:41.027737 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" podUID="8bf77172-40a2-4853-9e6a-02a0ed2cbc4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 01 05:59:42.028003 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:42.027966 5040 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 01 05:59:42.028407 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:42.028023 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" podUID="8bf77172-40a2-4853-9e6a-02a0ed2cbc4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 01 05:59:43.028084 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:43.028049 5040 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 01 05:59:43.028529 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:43.028112 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" podUID="8bf77172-40a2-4853-9e6a-02a0ed2cbc4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 01 05:59:44.031808 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:44.031779 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" Nov 01 05:59:44.032216 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:44.032194 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" Nov 01 05:59:44.032785 ip-10-0-143-195 kubenswrapper[5040]: I1101 05:59:44.032768 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" Nov 01 06:00:00.173725 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:00.173687 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891880-c7qgw"] Nov 01 06:00:00.178757 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:00.178731 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891880-9gw2c"] Nov 01 06:00:00.178880 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:00.178847 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891880-c7qgw" Nov 01 06:00:00.182221 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:00.182203 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891880-9gw2c" Nov 01 06:00:00.229924 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:00.229884 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wz9q\" (UniqueName: \"kubernetes.io/projected/b4288c33-5114-44e7-bc77-d311de2d7462-kube-api-access-5wz9q\") pod \"maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891880-c7qgw\" (UID: \"b4288c33-5114-44e7-bc77-d311de2d7462\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891880-c7qgw" Nov 01 06:00:00.229924 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:00.229938 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4288c33-5114-44e7-bc77-d311de2d7462-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891880-c7qgw\" (UID: \"b4288c33-5114-44e7-bc77-d311de2d7462\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891880-c7qgw" Nov 01 06:00:00.230155 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:00.230047 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ee39042-6210-456c-a1cf-9beaaf8dd33b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891880-9gw2c\" (UID: \"3ee39042-6210-456c-a1cf-9beaaf8dd33b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891880-9gw2c" Nov 01 06:00:00.230155 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:00.230085 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdkfb\" (UniqueName: \"kubernetes.io/projected/3ee39042-6210-456c-a1cf-9beaaf8dd33b-kube-api-access-hdkfb\") pod \"maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891880-9gw2c\" (UID: \"3ee39042-6210-456c-a1cf-9beaaf8dd33b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891880-9gw2c" Nov 01 06:00:00.231957 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:00.231936 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:00:00.232065 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:00.232011 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:00:00.232065 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:00.232016 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:00:00.285597 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:00.285549 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891880-9gw2c"] Nov 01 06:00:00.287894 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:00.287865 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891880-c7qgw"] Nov 01 06:00:00.331733 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:00.331700 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wz9q\" (UniqueName: \"kubernetes.io/projected/b4288c33-5114-44e7-bc77-d311de2d7462-kube-api-access-5wz9q\") pod \"maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891880-c7qgw\" (UID: \"b4288c33-5114-44e7-bc77-d311de2d7462\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891880-c7qgw" Nov 01 06:00:00.331733 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:00.331736 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4288c33-5114-44e7-bc77-d311de2d7462-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891880-c7qgw\" (UID: \"b4288c33-5114-44e7-bc77-d311de2d7462\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891880-c7qgw" Nov 01 06:00:00.331946 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:00.331770 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ee39042-6210-456c-a1cf-9beaaf8dd33b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891880-9gw2c\" (UID: \"3ee39042-6210-456c-a1cf-9beaaf8dd33b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891880-9gw2c" Nov 01 06:00:00.331946 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:00.331800 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdkfb\" (UniqueName: \"kubernetes.io/projected/3ee39042-6210-456c-a1cf-9beaaf8dd33b-kube-api-access-hdkfb\") pod \"maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891880-9gw2c\" (UID: \"3ee39042-6210-456c-a1cf-9beaaf8dd33b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891880-9gw2c" Nov 01 06:00:00.332504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:00.332458 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ee39042-6210-456c-a1cf-9beaaf8dd33b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891880-9gw2c\" (UID: \"3ee39042-6210-456c-a1cf-9beaaf8dd33b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891880-9gw2c" Nov 01 06:00:00.332596 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:00.332459 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4288c33-5114-44e7-bc77-d311de2d7462-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891880-c7qgw\" (UID: \"b4288c33-5114-44e7-bc77-d311de2d7462\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891880-c7qgw" Nov 01 06:00:00.394183 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:00.394146 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wz9q\" (UniqueName: \"kubernetes.io/projected/b4288c33-5114-44e7-bc77-d311de2d7462-kube-api-access-5wz9q\") pod \"maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891880-c7qgw\" (UID: \"b4288c33-5114-44e7-bc77-d311de2d7462\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891880-c7qgw" Nov 01 06:00:00.394455 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:00.394423 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdkfb\" (UniqueName: \"kubernetes.io/projected/3ee39042-6210-456c-a1cf-9beaaf8dd33b-kube-api-access-hdkfb\") pod \"maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891880-9gw2c\" (UID: \"3ee39042-6210-456c-a1cf-9beaaf8dd33b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891880-9gw2c" Nov 01 06:00:00.493447 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:00.493350 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891880-c7qgw" Nov 01 06:00:00.499758 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:00.499734 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891880-9gw2c" Nov 01 06:00:00.893342 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:00.893314 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891880-c7qgw"] Nov 01 06:00:00.894279 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:00:00.894245 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4288c33_5114_44e7_bc77_d311de2d7462.slice/crio-cfd992293b8e40910f80d296b4e620ae5ad4cde4a130d10dcad2c775c00823df WatchSource:0}: Error finding container cfd992293b8e40910f80d296b4e620ae5ad4cde4a130d10dcad2c775c00823df: Status 404 returned error can't find the container with id cfd992293b8e40910f80d296b4e620ae5ad4cde4a130d10dcad2c775c00823df Nov 01 06:00:00.942221 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:00.942186 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891880-9gw2c"] Nov 01 06:00:00.943592 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:00:00.943564 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ee39042_6210_456c_a1cf_9beaaf8dd33b.slice/crio-bb26ce170a8e1c738d5832345f9862ca521434117f65d2e8a218f9a3eecd13c0 WatchSource:0}: Error finding container bb26ce170a8e1c738d5832345f9862ca521434117f65d2e8a218f9a3eecd13c0: Status 404 returned error can't find the container with id bb26ce170a8e1c738d5832345f9862ca521434117f65d2e8a218f9a3eecd13c0 Nov 01 06:00:01.227725 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:01.227639 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891880-c7qgw" event={"ID":"b4288c33-5114-44e7-bc77-d311de2d7462","Type":"ContainerStarted","Data":"cfd992293b8e40910f80d296b4e620ae5ad4cde4a130d10dcad2c775c00823df"} Nov 01 06:00:01.229006 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:01.228981 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891880-9gw2c" event={"ID":"3ee39042-6210-456c-a1cf-9beaaf8dd33b","Type":"ContainerStarted","Data":"bb26ce170a8e1c738d5832345f9862ca521434117f65d2e8a218f9a3eecd13c0"} Nov 01 06:00:06.267694 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:06.267654 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891880-9gw2c" event={"ID":"3ee39042-6210-456c-a1cf-9beaaf8dd33b","Type":"ContainerStarted","Data":"e151ceae793d3374023c68871b38312a8f1d97f1408ff8464f7652ec3e1a5166"} Nov 01 06:00:06.269232 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:06.269209 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891880-c7qgw" event={"ID":"b4288c33-5114-44e7-bc77-d311de2d7462","Type":"ContainerStarted","Data":"06fb633c3324d90cdc9e290c21cba11af497b91920fc1aad6b213489ef745bf7"} Nov 01 06:00:06.385711 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:06.385605 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891880-9gw2c" podStartSLOduration=1.291814138 podStartE2EDuration="6.385587597s" totalImagesPullingTime="5.093773459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:00:06.383989933 +0000 UTC m=+885.039107100" watchObservedRunningTime="2026-11-01 06:00:06.385587597 +0000 UTC m=+885.040704765" Nov 01 06:00:06.501902 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:06.501851 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891880-c7qgw" podStartSLOduration=1.925518632 podStartE2EDuration="6.501838743s" totalImagesPullingTime="4.576320111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:00:06.500973091 +0000 UTC m=+885.156090270" watchObservedRunningTime="2026-11-01 06:00:06.501838743 +0000 UTC m=+885.156955910" Nov 01 06:00:06.932665 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:06.932626 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x"] Nov 01 06:00:06.936460 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:06.936443 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" Nov 01 06:00:06.994447 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:06.994353 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x"] Nov 01 06:00:06.994787 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:06.994761 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57cf5e07-36de-48cf-b938-138484cea3a6-home\") pod \"premium-w2-14a6e4-kserve-57f7f854cc-l5v9x\" (UID: \"57cf5e07-36de-48cf-b938-138484cea3a6\") " pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" Nov 01 06:00:06.994873 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:06.994802 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57cf5e07-36de-48cf-b938-138484cea3a6-model-cache\") pod \"premium-w2-14a6e4-kserve-57f7f854cc-l5v9x\" (UID: \"57cf5e07-36de-48cf-b938-138484cea3a6\") " pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" Nov 01 06:00:06.994935 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:06.994886 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57cf5e07-36de-48cf-b938-138484cea3a6-dshm\") pod \"premium-w2-14a6e4-kserve-57f7f854cc-l5v9x\" (UID: \"57cf5e07-36de-48cf-b938-138484cea3a6\") " pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" Nov 01 06:00:06.994935 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:06.994930 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cf5e07-36de-48cf-b938-138484cea3a6-tmp-dir\") pod \"premium-w2-14a6e4-kserve-57f7f854cc-l5v9x\" (UID: \"57cf5e07-36de-48cf-b938-138484cea3a6\") " pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" Nov 01 06:00:06.995041 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:06.994948 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrsfs\" (UniqueName: \"kubernetes.io/projected/57cf5e07-36de-48cf-b938-138484cea3a6-kube-api-access-wrsfs\") pod \"premium-w2-14a6e4-kserve-57f7f854cc-l5v9x\" (UID: \"57cf5e07-36de-48cf-b938-138484cea3a6\") " pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" Nov 01 06:00:06.995099 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:06.995078 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57cf5e07-36de-48cf-b938-138484cea3a6-tls-certs\") pod \"premium-w2-14a6e4-kserve-57f7f854cc-l5v9x\" (UID: \"57cf5e07-36de-48cf-b938-138484cea3a6\") " pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" Nov 01 06:00:07.041856 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:07.041824 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-14a6e4\"/\"premium-w2-14a6e4-kserve-self-signed-certs\"" Nov 01 06:00:07.096567 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:07.096520 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cf5e07-36de-48cf-b938-138484cea3a6-tmp-dir\") pod \"premium-w2-14a6e4-kserve-57f7f854cc-l5v9x\" (UID: \"57cf5e07-36de-48cf-b938-138484cea3a6\") " pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" Nov 01 06:00:07.096567 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:07.096557 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrsfs\" (UniqueName: \"kubernetes.io/projected/57cf5e07-36de-48cf-b938-138484cea3a6-kube-api-access-wrsfs\") pod \"premium-w2-14a6e4-kserve-57f7f854cc-l5v9x\" (UID: \"57cf5e07-36de-48cf-b938-138484cea3a6\") " pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" Nov 01 06:00:07.096894 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:07.096607 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57cf5e07-36de-48cf-b938-138484cea3a6-tls-certs\") pod \"premium-w2-14a6e4-kserve-57f7f854cc-l5v9x\" (UID: \"57cf5e07-36de-48cf-b938-138484cea3a6\") " pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" Nov 01 06:00:07.096894 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:07.096669 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57cf5e07-36de-48cf-b938-138484cea3a6-home\") pod \"premium-w2-14a6e4-kserve-57f7f854cc-l5v9x\" (UID: \"57cf5e07-36de-48cf-b938-138484cea3a6\") " pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" Nov 01 06:00:07.096894 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:07.096703 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57cf5e07-36de-48cf-b938-138484cea3a6-model-cache\") pod \"premium-w2-14a6e4-kserve-57f7f854cc-l5v9x\" (UID: \"57cf5e07-36de-48cf-b938-138484cea3a6\") " pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" Nov 01 06:00:07.096894 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:07.096746 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57cf5e07-36de-48cf-b938-138484cea3a6-dshm\") pod \"premium-w2-14a6e4-kserve-57f7f854cc-l5v9x\" (UID: \"57cf5e07-36de-48cf-b938-138484cea3a6\") " pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" Nov 01 06:00:07.097100 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:07.097037 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cf5e07-36de-48cf-b938-138484cea3a6-tmp-dir\") pod \"premium-w2-14a6e4-kserve-57f7f854cc-l5v9x\" (UID: \"57cf5e07-36de-48cf-b938-138484cea3a6\") " pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" Nov 01 06:00:07.097100 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:07.097058 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57cf5e07-36de-48cf-b938-138484cea3a6-home\") pod \"premium-w2-14a6e4-kserve-57f7f854cc-l5v9x\" (UID: \"57cf5e07-36de-48cf-b938-138484cea3a6\") " pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" Nov 01 06:00:07.097100 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:07.097078 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57cf5e07-36de-48cf-b938-138484cea3a6-model-cache\") pod \"premium-w2-14a6e4-kserve-57f7f854cc-l5v9x\" (UID: \"57cf5e07-36de-48cf-b938-138484cea3a6\") " pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" Nov 01 06:00:07.098971 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:07.098954 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57cf5e07-36de-48cf-b938-138484cea3a6-dshm\") pod \"premium-w2-14a6e4-kserve-57f7f854cc-l5v9x\" (UID: \"57cf5e07-36de-48cf-b938-138484cea3a6\") " pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" Nov 01 06:00:07.099171 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:07.099136 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57cf5e07-36de-48cf-b938-138484cea3a6-tls-certs\") pod \"premium-w2-14a6e4-kserve-57f7f854cc-l5v9x\" (UID: \"57cf5e07-36de-48cf-b938-138484cea3a6\") " pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" Nov 01 06:00:07.160515 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:07.160445 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrsfs\" (UniqueName: \"kubernetes.io/projected/57cf5e07-36de-48cf-b938-138484cea3a6-kube-api-access-wrsfs\") pod \"premium-w2-14a6e4-kserve-57f7f854cc-l5v9x\" (UID: \"57cf5e07-36de-48cf-b938-138484cea3a6\") " pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" Nov 01 06:00:07.248343 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:07.248300 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" Nov 01 06:00:07.291965 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:07.291929 5040 generic.go:356] "Generic (PLEG): container finished" podID="b4288c33-5114-44e7-bc77-d311de2d7462" containerID="06fb633c3324d90cdc9e290c21cba11af497b91920fc1aad6b213489ef745bf7" exitCode=60 Nov 01 06:00:07.292712 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:07.292679 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891880-c7qgw" event={"ID":"b4288c33-5114-44e7-bc77-d311de2d7462","Type":"ContainerDied","Data":"06fb633c3324d90cdc9e290c21cba11af497b91920fc1aad6b213489ef745bf7"} Nov 01 06:00:07.293113 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:07.293093 5040 scope.go:122] "RemoveContainer" containerID="06fb633c3324d90cdc9e290c21cba11af497b91920fc1aad6b213489ef745bf7" Nov 01 06:00:07.310874 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:07.308718 5040 generic.go:356] "Generic (PLEG): container finished" podID="3ee39042-6210-456c-a1cf-9beaaf8dd33b" containerID="e151ceae793d3374023c68871b38312a8f1d97f1408ff8464f7652ec3e1a5166" exitCode=60 Nov 01 06:00:07.310874 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:07.308765 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891880-9gw2c" event={"ID":"3ee39042-6210-456c-a1cf-9beaaf8dd33b","Type":"ContainerDied","Data":"e151ceae793d3374023c68871b38312a8f1d97f1408ff8464f7652ec3e1a5166"} Nov 01 06:00:07.310874 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:07.309161 5040 scope.go:122] "RemoveContainer" containerID="e151ceae793d3374023c68871b38312a8f1d97f1408ff8464f7652ec3e1a5166" Nov 01 06:00:07.676505 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:07.676456 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x"] Nov 01 06:00:07.676679 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:00:07.676656 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57cf5e07_36de_48cf_b938_138484cea3a6.slice/crio-c5021fe7b3298ee7e4717ac0e47c205fcacc86e0ed884b8fbba67982924ec859 WatchSource:0}: Error finding container c5021fe7b3298ee7e4717ac0e47c205fcacc86e0ed884b8fbba67982924ec859: Status 404 returned error can't find the container with id c5021fe7b3298ee7e4717ac0e47c205fcacc86e0ed884b8fbba67982924ec859 Nov 01 06:00:08.315855 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:08.315829 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" event={"ID":"57cf5e07-36de-48cf-b938-138484cea3a6","Type":"ContainerStarted","Data":"7daac90a2fc5ccdcea8d31696984b7a7183171a108776bd09e43561ba3977eaa"} Nov 01 06:00:08.316188 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:08.315872 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" event={"ID":"57cf5e07-36de-48cf-b938-138484cea3a6","Type":"ContainerStarted","Data":"c5021fe7b3298ee7e4717ac0e47c205fcacc86e0ed884b8fbba67982924ec859"} Nov 01 06:00:08.317684 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:08.317663 5040 generic.go:356] "Generic (PLEG): container finished" podID="3ee39042-6210-456c-a1cf-9beaaf8dd33b" containerID="b13b8d17e997076655351f60e74703ec6badfa426b8dfcd26ae1faff6c9120bf" exitCode=60 Nov 01 06:00:08.317767 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:08.317743 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891880-9gw2c" event={"ID":"3ee39042-6210-456c-a1cf-9beaaf8dd33b","Type":"ContainerDied","Data":"b13b8d17e997076655351f60e74703ec6badfa426b8dfcd26ae1faff6c9120bf"} Nov 01 06:00:08.317815 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:08.317800 5040 scope.go:122] "RemoveContainer" containerID="e151ceae793d3374023c68871b38312a8f1d97f1408ff8464f7652ec3e1a5166" Nov 01 06:00:08.317973 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:08.317953 5040 scope.go:122] "RemoveContainer" containerID="b13b8d17e997076655351f60e74703ec6badfa426b8dfcd26ae1faff6c9120bf" Nov 01 06:00:08.318225 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:00:08.318201 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891880-9gw2c_odh-ai-gateway-infra(3ee39042-6210-456c-a1cf-9beaaf8dd33b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891880-9gw2c" podUID="3ee39042-6210-456c-a1cf-9beaaf8dd33b" Nov 01 06:00:08.319437 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:08.319415 5040 generic.go:356] "Generic (PLEG): container finished" podID="b4288c33-5114-44e7-bc77-d311de2d7462" containerID="ceb9ebaa4f4a78f0306427e6206183c5884cd4b6bbe7fc16f1d976163aa803ca" exitCode=60 Nov 01 06:00:08.319533 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:08.319438 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891880-c7qgw" event={"ID":"b4288c33-5114-44e7-bc77-d311de2d7462","Type":"ContainerDied","Data":"ceb9ebaa4f4a78f0306427e6206183c5884cd4b6bbe7fc16f1d976163aa803ca"} Nov 01 06:00:08.319723 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:08.319710 5040 scope.go:122] "RemoveContainer" containerID="ceb9ebaa4f4a78f0306427e6206183c5884cd4b6bbe7fc16f1d976163aa803ca" Nov 01 06:00:08.319905 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:00:08.319889 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891880-c7qgw_odh-ai-gateway-infra(b4288c33-5114-44e7-bc77-d311de2d7462)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891880-c7qgw" podUID="b4288c33-5114-44e7-bc77-d311de2d7462" Nov 01 06:00:08.418905 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:08.418883 5040 scope.go:122] "RemoveContainer" containerID="06fb633c3324d90cdc9e290c21cba11af497b91920fc1aad6b213489ef745bf7" Nov 01 06:00:08.436747 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:08.436704 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" podStartSLOduration=2.436692286 podStartE2EDuration="2.436692286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:00:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:00:08.434898138 +0000 UTC m=+887.090015307" watchObservedRunningTime="2026-11-01 06:00:08.436692286 +0000 UTC m=+887.091809452" Nov 01 06:00:09.248904 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:09.248868 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" Nov 01 06:00:09.326866 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:09.326830 5040 scope.go:122] "RemoveContainer" containerID="ceb9ebaa4f4a78f0306427e6206183c5884cd4b6bbe7fc16f1d976163aa803ca" Nov 01 06:00:09.327328 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:00:09.327088 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891880-c7qgw_odh-ai-gateway-infra(b4288c33-5114-44e7-bc77-d311de2d7462)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891880-c7qgw" podUID="b4288c33-5114-44e7-bc77-d311de2d7462" Nov 01 06:00:09.328493 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:09.328456 5040 scope.go:122] "RemoveContainer" containerID="b13b8d17e997076655351f60e74703ec6badfa426b8dfcd26ae1faff6c9120bf" Nov 01 06:00:09.328751 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:00:09.328732 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891880-9gw2c_odh-ai-gateway-infra(3ee39042-6210-456c-a1cf-9beaaf8dd33b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891880-9gw2c" podUID="3ee39042-6210-456c-a1cf-9beaaf8dd33b" Nov 01 06:00:09.972610 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:00:09.972568 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:00:17.249324 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:17.249287 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" Nov 01 06:00:17.263886 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:17.263861 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" Nov 01 06:00:17.392599 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:17.392567 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" Nov 01 06:00:19.507231 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:19.507203 5040 scope.go:122] "RemoveContainer" containerID="b13b8d17e997076655351f60e74703ec6badfa426b8dfcd26ae1faff6c9120bf" Nov 01 06:00:20.398092 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:20.398061 5040 generic.go:356] "Generic (PLEG): container finished" podID="3ee39042-6210-456c-a1cf-9beaaf8dd33b" containerID="a112b121075e71f30bcf2448cf0fd54310551635ac500aedceac657d24d13c7b" exitCode=60 Nov 01 06:00:20.398277 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:20.398130 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891880-9gw2c" event={"ID":"3ee39042-6210-456c-a1cf-9beaaf8dd33b","Type":"ContainerDied","Data":"a112b121075e71f30bcf2448cf0fd54310551635ac500aedceac657d24d13c7b"} Nov 01 06:00:20.398277 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:20.398189 5040 scope.go:122] "RemoveContainer" containerID="b13b8d17e997076655351f60e74703ec6badfa426b8dfcd26ae1faff6c9120bf" Nov 01 06:00:20.398614 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:20.398598 5040 scope.go:122] "RemoveContainer" containerID="a112b121075e71f30bcf2448cf0fd54310551635ac500aedceac657d24d13c7b" Nov 01 06:00:20.398873 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:00:20.398854 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891880-9gw2c_odh-ai-gateway-infra(3ee39042-6210-456c-a1cf-9beaaf8dd33b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891880-9gw2c" podUID="3ee39042-6210-456c-a1cf-9beaaf8dd33b" Nov 01 06:00:21.568621 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:21.568586 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891880-9gw2c"] Nov 01 06:00:21.602103 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:21.602070 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 06:00:21.602528 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:21.602509 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 06:00:21.722772 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:21.722750 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891880-9gw2c" Nov 01 06:00:21.858295 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:21.858220 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ee39042-6210-456c-a1cf-9beaaf8dd33b-kube-api-access-hdkfb\" (UniqueName: \"kubernetes.io/projected/3ee39042-6210-456c-a1cf-9beaaf8dd33b-kube-api-access-hdkfb\") pod \"3ee39042-6210-456c-a1cf-9beaaf8dd33b\" (UID: \"3ee39042-6210-456c-a1cf-9beaaf8dd33b\") " Nov 01 06:00:21.858439 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:21.858327 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ee39042-6210-456c-a1cf-9beaaf8dd33b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ee39042-6210-456c-a1cf-9beaaf8dd33b-maas-api-service-ca\") pod \"3ee39042-6210-456c-a1cf-9beaaf8dd33b\" (UID: \"3ee39042-6210-456c-a1cf-9beaaf8dd33b\") " Nov 01 06:00:21.858700 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:21.858678 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ee39042-6210-456c-a1cf-9beaaf8dd33b-maas-api-service-ca" pod "3ee39042-6210-456c-a1cf-9beaaf8dd33b" (UID: "3ee39042-6210-456c-a1cf-9beaaf8dd33b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:00:21.860804 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:21.860779 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ee39042-6210-456c-a1cf-9beaaf8dd33b-kube-api-access-hdkfb" pod "3ee39042-6210-456c-a1cf-9beaaf8dd33b" (UID: "3ee39042-6210-456c-a1cf-9beaaf8dd33b"). InnerVolumeSpecName "kube-api-access-hdkfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:00:21.959644 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:21.959616 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ee39042-6210-456c-a1cf-9beaaf8dd33b-maas-api-service-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:00:21.959644 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:21.959640 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdkfb\" (UniqueName: \"kubernetes.io/projected/3ee39042-6210-456c-a1cf-9beaaf8dd33b-kube-api-access-hdkfb\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:00:22.412092 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:22.412067 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891880-9gw2c" Nov 01 06:00:22.412092 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:22.412070 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891880-9gw2c" event={"ID":"3ee39042-6210-456c-a1cf-9beaaf8dd33b","Type":"ContainerDied","Data":"bb26ce170a8e1c738d5832345f9862ca521434117f65d2e8a218f9a3eecd13c0"} Nov 01 06:00:22.412339 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:22.412131 5040 scope.go:122] "RemoveContainer" containerID="a112b121075e71f30bcf2448cf0fd54310551635ac500aedceac657d24d13c7b" Nov 01 06:00:22.507680 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:22.507651 5040 scope.go:122] "RemoveContainer" containerID="ceb9ebaa4f4a78f0306427e6206183c5884cd4b6bbe7fc16f1d976163aa803ca" Nov 01 06:00:22.580223 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:22.580198 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891880-9gw2c"] Nov 01 06:00:22.588475 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:22.588452 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891880-9gw2c"] Nov 01 06:00:23.426991 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:23.426955 5040 generic.go:356] "Generic (PLEG): container finished" podID="b4288c33-5114-44e7-bc77-d311de2d7462" containerID="4a80cf96601bc5bd9a5ce54e02e35a9d2da2b1a03b7f4752766f1229b5b0ec91" exitCode=60 Nov 01 06:00:23.426991 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:23.426994 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891880-c7qgw" event={"ID":"b4288c33-5114-44e7-bc77-d311de2d7462","Type":"ContainerDied","Data":"4a80cf96601bc5bd9a5ce54e02e35a9d2da2b1a03b7f4752766f1229b5b0ec91"} Nov 01 06:00:23.427211 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:23.427059 5040 scope.go:122] "RemoveContainer" containerID="ceb9ebaa4f4a78f0306427e6206183c5884cd4b6bbe7fc16f1d976163aa803ca" Nov 01 06:00:23.427395 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:23.427376 5040 scope.go:122] "RemoveContainer" containerID="4a80cf96601bc5bd9a5ce54e02e35a9d2da2b1a03b7f4752766f1229b5b0ec91" Nov 01 06:00:23.427665 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:00:23.427645 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891880-c7qgw_odh-ai-gateway-infra(b4288c33-5114-44e7-bc77-d311de2d7462)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891880-c7qgw" podUID="b4288c33-5114-44e7-bc77-d311de2d7462" Nov 01 06:00:23.515326 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:23.515297 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ee39042-6210-456c-a1cf-9beaaf8dd33b" path="/var/lib/kubelet/pods/3ee39042-6210-456c-a1cf-9beaaf8dd33b/volumes" Nov 01 06:00:24.554046 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:24.554009 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891880-c7qgw"] Nov 01 06:00:24.697701 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:24.697678 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891880-c7qgw" Nov 01 06:00:24.787018 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:24.786990 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4288c33-5114-44e7-bc77-d311de2d7462-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4288c33-5114-44e7-bc77-d311de2d7462-maas-api-service-ca\") pod \"b4288c33-5114-44e7-bc77-d311de2d7462\" (UID: \"b4288c33-5114-44e7-bc77-d311de2d7462\") " Nov 01 06:00:24.787179 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:24.787111 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4288c33-5114-44e7-bc77-d311de2d7462-kube-api-access-5wz9q\" (UniqueName: \"kubernetes.io/projected/b4288c33-5114-44e7-bc77-d311de2d7462-kube-api-access-5wz9q\") pod \"b4288c33-5114-44e7-bc77-d311de2d7462\" (UID: \"b4288c33-5114-44e7-bc77-d311de2d7462\") " Nov 01 06:00:24.787348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:24.787322 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4288c33-5114-44e7-bc77-d311de2d7462-maas-api-service-ca" pod "b4288c33-5114-44e7-bc77-d311de2d7462" (UID: "b4288c33-5114-44e7-bc77-d311de2d7462"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:00:24.789187 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:24.789160 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4288c33-5114-44e7-bc77-d311de2d7462-kube-api-access-5wz9q" pod "b4288c33-5114-44e7-bc77-d311de2d7462" (UID: "b4288c33-5114-44e7-bc77-d311de2d7462"). InnerVolumeSpecName "kube-api-access-5wz9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:00:24.887993 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:24.887915 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wz9q\" (UniqueName: \"kubernetes.io/projected/b4288c33-5114-44e7-bc77-d311de2d7462-kube-api-access-5wz9q\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:00:24.887993 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:24.887944 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4288c33-5114-44e7-bc77-d311de2d7462-maas-api-service-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:00:25.440594 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:25.440559 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891880-c7qgw" event={"ID":"b4288c33-5114-44e7-bc77-d311de2d7462","Type":"ContainerDied","Data":"cfd992293b8e40910f80d296b4e620ae5ad4cde4a130d10dcad2c775c00823df"} Nov 01 06:00:25.440780 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:25.440613 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891880-c7qgw" Nov 01 06:00:25.440780 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:25.440644 5040 scope.go:122] "RemoveContainer" containerID="4a80cf96601bc5bd9a5ce54e02e35a9d2da2b1a03b7f4752766f1229b5b0ec91" Nov 01 06:00:25.612257 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:25.612229 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891880-c7qgw"] Nov 01 06:00:25.663023 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:25.662991 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891880-c7qgw"] Nov 01 06:00:27.518091 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:27.518049 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4288c33-5114-44e7-bc77-d311de2d7462" path="/var/lib/kubelet/pods/b4288c33-5114-44e7-bc77-d311de2d7462/volumes" Nov 01 06:00:39.972724 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:00:39.972685 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:00:47.182401 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.182365 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj"] Nov 01 06:00:47.182945 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.182828 5040 cpu_manager.go:414] "containerMap: removing container" podUID="3ee39042-6210-456c-a1cf-9beaaf8dd33b" containerName="cleanup" Nov 01 06:00:47.182945 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.182842 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ee39042-6210-456c-a1cf-9beaaf8dd33b" containerName="cleanup" Nov 01 06:00:47.182945 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.182850 5040 cpu_manager.go:414] "containerMap: removing container" podUID="b4288c33-5114-44e7-bc77-d311de2d7462" containerName="cleanup" Nov 01 06:00:47.182945 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.182854 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4288c33-5114-44e7-bc77-d311de2d7462" containerName="cleanup" Nov 01 06:00:47.182945 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.182867 5040 cpu_manager.go:414] "containerMap: removing container" podUID="3ee39042-6210-456c-a1cf-9beaaf8dd33b" containerName="cleanup" Nov 01 06:00:47.182945 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.182872 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ee39042-6210-456c-a1cf-9beaaf8dd33b" containerName="cleanup" Nov 01 06:00:47.182945 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.182887 5040 cpu_manager.go:414] "containerMap: removing container" podUID="b4288c33-5114-44e7-bc77-d311de2d7462" containerName="cleanup" Nov 01 06:00:47.182945 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.182893 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4288c33-5114-44e7-bc77-d311de2d7462" containerName="cleanup" Nov 01 06:00:47.183188 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.182956 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4288c33-5114-44e7-bc77-d311de2d7462" containerName="cleanup" Nov 01 06:00:47.183188 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.182963 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4288c33-5114-44e7-bc77-d311de2d7462" containerName="cleanup" Nov 01 06:00:47.183188 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.182968 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ee39042-6210-456c-a1cf-9beaaf8dd33b" containerName="cleanup" Nov 01 06:00:47.183188 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.182978 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ee39042-6210-456c-a1cf-9beaaf8dd33b" containerName="cleanup" Nov 01 06:00:47.192969 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.192942 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" Nov 01 06:00:47.246346 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.246315 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-14a6e4-openshift-default-dockercfg-vxhc4\"" Nov 01 06:00:47.293183 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.293141 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-istio-podinfo\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj\" (UID: \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" Nov 01 06:00:47.293357 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.293278 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpx2d\" (UniqueName: \"kubernetes.io/projected/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-kube-api-access-jpx2d\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj\" (UID: \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" Nov 01 06:00:47.293357 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.293334 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-workload-certs\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj\" (UID: \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" Nov 01 06:00:47.293497 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.293466 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-istio-data\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj\" (UID: \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" Nov 01 06:00:47.293602 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.293581 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-credential-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj\" (UID: \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" Nov 01 06:00:47.293665 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.293636 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-workload-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj\" (UID: \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" Nov 01 06:00:47.293724 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.293665 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-istiod-ca-cert\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj\" (UID: \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" Nov 01 06:00:47.293724 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.293692 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-istio-token\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj\" (UID: \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" Nov 01 06:00:47.293827 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.293725 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-istio-envoy\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj\" (UID: \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" Nov 01 06:00:47.299450 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.299424 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj"] Nov 01 06:00:47.394743 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.394708 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-credential-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj\" (UID: \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" Nov 01 06:00:47.394939 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.394751 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-workload-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj\" (UID: \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" Nov 01 06:00:47.394939 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.394770 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-istiod-ca-cert\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj\" (UID: \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" Nov 01 06:00:47.394939 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.394786 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-istio-token\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj\" (UID: \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" Nov 01 06:00:47.394939 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.394816 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-istio-envoy\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj\" (UID: \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" Nov 01 06:00:47.394939 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.394845 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-istio-podinfo\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj\" (UID: \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" Nov 01 06:00:47.395210 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.394990 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpx2d\" (UniqueName: \"kubernetes.io/projected/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-kube-api-access-jpx2d\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj\" (UID: \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" Nov 01 06:00:47.395210 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.395034 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-workload-certs\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj\" (UID: \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" Nov 01 06:00:47.395210 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.395125 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-istio-data\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj\" (UID: \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" Nov 01 06:00:47.395365 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.395211 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-workload-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj\" (UID: \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" Nov 01 06:00:47.395365 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.395303 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-credential-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj\" (UID: \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" Nov 01 06:00:47.395471 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.395450 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-istio-data\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj\" (UID: \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" Nov 01 06:00:47.395606 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.395590 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-istiod-ca-cert\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj\" (UID: \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" Nov 01 06:00:47.395695 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.395678 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-workload-certs\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj\" (UID: \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" Nov 01 06:00:47.397760 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.397737 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-istio-envoy\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj\" (UID: \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" Nov 01 06:00:47.397896 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.397879 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-istio-podinfo\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj\" (UID: \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" Nov 01 06:00:47.457104 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.457036 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-istio-token\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj\" (UID: \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" Nov 01 06:00:47.457104 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.457081 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpx2d\" (UniqueName: \"kubernetes.io/projected/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-kube-api-access-jpx2d\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj\" (UID: \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" Nov 01 06:00:47.506824 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.506794 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" Nov 01 06:00:47.936589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.936562 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj"] Nov 01 06:00:47.937450 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:00:47.937426 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbafbb5e_f923_447d_a4ac_8a0d4ef73752.slice/crio-f7e69d2f40be441d5095504761be0b53c01f5ef13226315835f486fc49384750 WatchSource:0}: Error finding container f7e69d2f40be441d5095504761be0b53c01f5ef13226315835f486fc49384750: Status 404 returned error can't find the container with id f7e69d2f40be441d5095504761be0b53c01f5ef13226315835f486fc49384750 Nov 01 06:00:47.939760 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.939728 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:00:47.939836 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.939794 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:00:47.939836 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:47.939825 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:00:48.577894 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:48.577839 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" event={"ID":"bbafbb5e-f923-447d-a4ac-8a0d4ef73752","Type":"ContainerStarted","Data":"10a0e62631b7b24f0983b45edf83d24db1fd3a76ccf270267a75b565fcce06f5"} Nov 01 06:00:48.578395 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:48.577916 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" event={"ID":"bbafbb5e-f923-447d-a4ac-8a0d4ef73752","Type":"ContainerStarted","Data":"f7e69d2f40be441d5095504761be0b53c01f5ef13226315835f486fc49384750"} Nov 01 06:00:48.702501 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:48.701943 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" podStartSLOduration=1.701924498 podStartE2EDuration="1.701924498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:00:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:00:48.701104243 +0000 UTC m=+927.356221434" watchObservedRunningTime="2026-11-01 06:00:48.701924498 +0000 UTC m=+927.357041662" Nov 01 06:00:49.511501 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:49.507820 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" Nov 01 06:00:49.511704 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:49.511581 5040 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 01 06:00:49.511704 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:49.511649 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" podUID="bbafbb5e-f923-447d-a4ac-8a0d4ef73752" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 01 06:00:50.512853 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:50.512817 5040 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 01 06:00:50.514590 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:50.514557 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" podUID="bbafbb5e-f923-447d-a4ac-8a0d4ef73752" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 01 06:00:51.513944 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:51.513885 5040 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 01 06:00:51.514552 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:51.514519 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" podUID="bbafbb5e-f923-447d-a4ac-8a0d4ef73752" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 01 06:00:52.510247 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:52.509789 5040 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 01 06:00:52.510247 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:52.509870 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" podUID="bbafbb5e-f923-447d-a4ac-8a0d4ef73752" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 01 06:00:53.514897 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:53.514852 5040 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 01 06:00:53.515504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:53.515450 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" podUID="bbafbb5e-f923-447d-a4ac-8a0d4ef73752" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 01 06:00:54.531634 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:54.531597 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" Nov 01 06:00:54.532552 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:54.532531 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" Nov 01 06:00:54.533112 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:00:54.533093 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" Nov 01 06:01:09.972563 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:01:09.972472 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:01:39.972752 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:01:39.972717 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:01:40.895672 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:40.895640 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7125-9b2915"] Nov 01 06:01:40.896062 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:40.896049 5040 cpu_manager.go:414] "containerMap: removing container" podUID="3ee39042-6210-456c-a1cf-9beaaf8dd33b" containerName="cleanup" Nov 01 06:01:40.896062 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:40.896062 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ee39042-6210-456c-a1cf-9beaaf8dd33b" containerName="cleanup" Nov 01 06:01:40.896187 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:40.896082 5040 cpu_manager.go:414] "containerMap: removing container" podUID="b4288c33-5114-44e7-bc77-d311de2d7462" containerName="cleanup" Nov 01 06:01:40.896187 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:40.896087 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4288c33-5114-44e7-bc77-d311de2d7462" containerName="cleanup" Nov 01 06:01:40.896262 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:40.896188 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ee39042-6210-456c-a1cf-9beaaf8dd33b" containerName="cleanup" Nov 01 06:01:40.896262 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:40.896198 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4288c33-5114-44e7-bc77-d311de2d7462" containerName="cleanup" Nov 01 06:01:40.899228 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:40.899209 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7125-9b2915" Nov 01 06:01:40.937034 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:40.937004 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmcwq\" (UniqueName: \"kubernetes.io/projected/9438692f-ad16-4b54-952b-ef153b44e25f-kube-api-access-xmcwq\") pod \"test-curl-7125-9b2915\" (UID: \"9438692f-ad16-4b54-952b-ef153b44e25f\") " pod="opendatahub/test-curl-7125-9b2915" Nov 01 06:01:40.954987 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:40.954955 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7125-9b2915"] Nov 01 06:01:41.002208 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:41.002174 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hgn5l\"" Nov 01 06:01:41.038566 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:41.038534 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmcwq\" (UniqueName: \"kubernetes.io/projected/9438692f-ad16-4b54-952b-ef153b44e25f-kube-api-access-xmcwq\") pod \"test-curl-7125-9b2915\" (UID: \"9438692f-ad16-4b54-952b-ef153b44e25f\") " pod="opendatahub/test-curl-7125-9b2915" Nov 01 06:01:41.100028 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:41.100000 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmcwq\" (UniqueName: \"kubernetes.io/projected/9438692f-ad16-4b54-952b-ef153b44e25f-kube-api-access-xmcwq\") pod \"test-curl-7125-9b2915\" (UID: \"9438692f-ad16-4b54-952b-ef153b44e25f\") " pod="opendatahub/test-curl-7125-9b2915" Nov 01 06:01:41.211339 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:41.211253 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7125-9b2915" Nov 01 06:01:41.583534 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:41.583498 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7125-9b2915"] Nov 01 06:01:41.585008 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:01:41.584983 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9438692f_ad16_4b54_952b_ef153b44e25f.slice/crio-21af1597debf5f3f990cf0dd8ca31c1c988aa93968e4494cc463f0a5ff870d95 WatchSource:0}: Error finding container 21af1597debf5f3f990cf0dd8ca31c1c988aa93968e4494cc463f0a5ff870d95: Status 404 returned error can't find the container with id 21af1597debf5f3f990cf0dd8ca31c1c988aa93968e4494cc463f0a5ff870d95 Nov 01 06:01:41.586884 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:41.586864 5040 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 06:01:41.907783 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:41.907697 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7125-9b2915" event={"ID":"9438692f-ad16-4b54-952b-ef153b44e25f","Type":"ContainerStarted","Data":"21af1597debf5f3f990cf0dd8ca31c1c988aa93968e4494cc463f0a5ff870d95"} Nov 01 06:01:42.919564 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:42.919510 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7125-9b2915" event={"ID":"9438692f-ad16-4b54-952b-ef153b44e25f","Type":"ContainerStarted","Data":"9d7e62271d1b4ce6cf376969c960fb78fedbea8561ae2107de7c04c005764e6f"} Nov 01 06:01:43.034427 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:43.034376 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7125-9b2915" podStartSLOduration=1.910447309 podStartE2EDuration="3.03436262s" totalImagesPullingTime="1.123915311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:01:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:01:43.034030922 +0000 UTC m=+981.689148088" watchObservedRunningTime="2026-11-01 06:01:43.03436262 +0000 UTC m=+981.689479790" Nov 01 06:01:44.459033 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:44.458992 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7125-9b2915"] Nov 01 06:01:44.463740 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:44.463708 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7125-9b2915"] Nov 01 06:01:44.627095 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:44.627060 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7125-c7efa5"] Nov 01 06:01:44.627506 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:44.627466 5040 cpu_manager.go:414] "containerMap: removing container" podUID="9438692f-ad16-4b54-952b-ef153b44e25f" containerName="test-curl-7125-9b2915" Nov 01 06:01:44.627506 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:44.627492 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9438692f-ad16-4b54-952b-ef153b44e25f" containerName="test-curl-7125-9b2915" Nov 01 06:01:44.627613 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:44.627603 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="9438692f-ad16-4b54-952b-ef153b44e25f" containerName="test-curl-7125-9b2915" Nov 01 06:01:44.630841 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:44.630817 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7125-c7efa5" Nov 01 06:01:44.673475 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:44.673444 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xmtm\" (UniqueName: \"kubernetes.io/projected/ec716a4b-ce8d-46c0-a9f3-864253d1de3e-kube-api-access-4xmtm\") pod \"test-curl-7125-c7efa5\" (UID: \"ec716a4b-ce8d-46c0-a9f3-864253d1de3e\") " pod="opendatahub/test-curl-7125-c7efa5" Nov 01 06:01:44.683871 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:44.683844 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7125-c7efa5"] Nov 01 06:01:44.775006 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:44.774972 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xmtm\" (UniqueName: \"kubernetes.io/projected/ec716a4b-ce8d-46c0-a9f3-864253d1de3e-kube-api-access-4xmtm\") pod \"test-curl-7125-c7efa5\" (UID: \"ec716a4b-ce8d-46c0-a9f3-864253d1de3e\") " pod="opendatahub/test-curl-7125-c7efa5" Nov 01 06:01:44.836834 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:44.836800 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xmtm\" (UniqueName: \"kubernetes.io/projected/ec716a4b-ce8d-46c0-a9f3-864253d1de3e-kube-api-access-4xmtm\") pod \"test-curl-7125-c7efa5\" (UID: \"ec716a4b-ce8d-46c0-a9f3-864253d1de3e\") " pod="opendatahub/test-curl-7125-c7efa5" Nov 01 06:01:44.930934 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:44.930894 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7125-9b2915" podUID="9438692f-ad16-4b54-952b-ef153b44e25f" containerName="test-curl-7125-9b2915" containerID="cri-o://9d7e62271d1b4ce6cf376969c960fb78fedbea8561ae2107de7c04c005764e6f" gracePeriod=2 Nov 01 06:01:44.943423 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:44.943397 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7125-c7efa5" Nov 01 06:01:44.982969 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:01:44.982921 5040 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7125-9b2915\" is forbidden: User \"system:node:ip-10-0-143-195.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-195.us-west-1.compute.internal' and this object" podUID="9438692f-ad16-4b54-952b-ef153b44e25f" pod="opendatahub/test-curl-7125-9b2915" Nov 01 06:01:45.315844 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:45.315820 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7125-c7efa5"] Nov 01 06:01:45.317291 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:01:45.317257 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec716a4b_ce8d_46c0_a9f3_864253d1de3e.slice/crio-d7f730a0db66da9be47482df025cdad0a094d02bc2a75ce73cd47884181d731c WatchSource:0}: Error finding container d7f730a0db66da9be47482df025cdad0a094d02bc2a75ce73cd47884181d731c: Status 404 returned error can't find the container with id d7f730a0db66da9be47482df025cdad0a094d02bc2a75ce73cd47884181d731c Nov 01 06:01:45.937205 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:45.937172 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7125-c7efa5" event={"ID":"ec716a4b-ce8d-46c0-a9f3-864253d1de3e","Type":"ContainerStarted","Data":"31b3ac3b17a4232718103cd93d6f75de953a6166ab58065ddbbb9e2d37ee7bfd"} Nov 01 06:01:45.937621 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:45.937217 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7125-c7efa5" event={"ID":"ec716a4b-ce8d-46c0-a9f3-864253d1de3e","Type":"ContainerStarted","Data":"d7f730a0db66da9be47482df025cdad0a094d02bc2a75ce73cd47884181d731c"} Nov 01 06:01:46.079505 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:46.075864 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7125-c7efa5" podStartSLOduration=2.075844147 podStartE2EDuration="2.075844147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:01:46.063776031 +0000 UTC m=+984.718893198" watchObservedRunningTime="2026-11-01 06:01:46.075844147 +0000 UTC m=+984.730961315" Nov 01 06:01:47.179318 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:47.179287 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7125-9b2915" Nov 01 06:01:47.303271 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:47.303233 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9438692f-ad16-4b54-952b-ef153b44e25f-kube-api-access-xmcwq\" (UniqueName: \"kubernetes.io/projected/9438692f-ad16-4b54-952b-ef153b44e25f-kube-api-access-xmcwq\") pod \"9438692f-ad16-4b54-952b-ef153b44e25f\" (UID: \"9438692f-ad16-4b54-952b-ef153b44e25f\") " Nov 01 06:01:47.305327 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:47.305298 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9438692f-ad16-4b54-952b-ef153b44e25f-kube-api-access-xmcwq" pod "9438692f-ad16-4b54-952b-ef153b44e25f" (UID: "9438692f-ad16-4b54-952b-ef153b44e25f"). InnerVolumeSpecName "kube-api-access-xmcwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:01:47.404955 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:47.404919 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmcwq\" (UniqueName: \"kubernetes.io/projected/9438692f-ad16-4b54-952b-ef153b44e25f-kube-api-access-xmcwq\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:01:47.515612 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:47.515579 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9438692f-ad16-4b54-952b-ef153b44e25f" path="/var/lib/kubelet/pods/9438692f-ad16-4b54-952b-ef153b44e25f/volumes" Nov 01 06:01:47.578645 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:47.578559 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7125-c7efa5"] Nov 01 06:01:47.582848 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:47.582813 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7125-c7efa5"] Nov 01 06:01:47.949793 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:47.949694 5040 generic.go:356] "Generic (PLEG): container finished" podID="9438692f-ad16-4b54-952b-ef153b44e25f" containerID="9d7e62271d1b4ce6cf376969c960fb78fedbea8561ae2107de7c04c005764e6f" exitCode=137 Nov 01 06:01:47.949793 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:47.949746 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7125-9b2915" Nov 01 06:01:47.949793 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:47.949784 5040 scope.go:122] "RemoveContainer" containerID="9d7e62271d1b4ce6cf376969c960fb78fedbea8561ae2107de7c04c005764e6f" Nov 01 06:01:47.950071 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:47.949964 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7125-c7efa5" podUID="ec716a4b-ce8d-46c0-a9f3-864253d1de3e" containerName="test-curl-7125-c7efa5" containerID="cri-o://31b3ac3b17a4232718103cd93d6f75de953a6166ab58065ddbbb9e2d37ee7bfd" gracePeriod=2 Nov 01 06:01:47.958873 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:47.958854 5040 scope.go:122] "RemoveContainer" containerID="9d7e62271d1b4ce6cf376969c960fb78fedbea8561ae2107de7c04c005764e6f" Nov 01 06:01:47.959154 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:01:47.959135 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d7e62271d1b4ce6cf376969c960fb78fedbea8561ae2107de7c04c005764e6f\": container with ID starting with 9d7e62271d1b4ce6cf376969c960fb78fedbea8561ae2107de7c04c005764e6f not found: ID does not exist" containerID="9d7e62271d1b4ce6cf376969c960fb78fedbea8561ae2107de7c04c005764e6f" Nov 01 06:01:47.959209 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:47.959168 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d7e62271d1b4ce6cf376969c960fb78fedbea8561ae2107de7c04c005764e6f"} err="failed to get container status \"9d7e62271d1b4ce6cf376969c960fb78fedbea8561ae2107de7c04c005764e6f\": rpc error: code = NotFound desc = could not find container \"9d7e62271d1b4ce6cf376969c960fb78fedbea8561ae2107de7c04c005764e6f\": container with ID starting with 9d7e62271d1b4ce6cf376969c960fb78fedbea8561ae2107de7c04c005764e6f not found: ID does not exist" Nov 01 06:01:48.002421 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:01:48.002382 5040 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7125-c7efa5\" is forbidden: User \"system:node:ip-10-0-143-195.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-195.us-west-1.compute.internal' and this object" podUID="ec716a4b-ce8d-46c0-a9f3-864253d1de3e" pod="opendatahub/test-curl-7125-c7efa5" Nov 01 06:01:48.054804 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:01:48.054769 5040 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7125-c7efa5\" is forbidden: User \"system:node:ip-10-0-143-195.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-195.us-west-1.compute.internal' and this object" podUID="ec716a4b-ce8d-46c0-a9f3-864253d1de3e" pod="opendatahub/test-curl-7125-c7efa5" Nov 01 06:01:49.955871 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:49.955839 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2"] Nov 01 06:01:49.956351 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:49.956333 5040 cpu_manager.go:414] "containerMap: removing container" podUID="ec716a4b-ce8d-46c0-a9f3-864253d1de3e" containerName="test-curl-7125-c7efa5" Nov 01 06:01:49.956419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:49.956351 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec716a4b-ce8d-46c0-a9f3-864253d1de3e" containerName="test-curl-7125-c7efa5" Nov 01 06:01:49.956508 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:49.956493 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec716a4b-ce8d-46c0-a9f3-864253d1de3e" containerName="test-curl-7125-c7efa5" Nov 01 06:01:49.970253 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:49.970225 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:01:50.026727 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.026690 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-cfe368-openshift-default-dockercfg-9lj8x\"" Nov 01 06:01:50.049263 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.049223 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-istio-envoy\") pod \"e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2\" (UID: \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\") " pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:01:50.049454 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.049275 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-istio-data\") pod \"e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2\" (UID: \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\") " pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:01:50.049454 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.049339 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52bj2\" (UniqueName: \"kubernetes.io/projected/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-kube-api-access-52bj2\") pod \"e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2\" (UID: \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\") " pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:01:50.049454 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.049369 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-credential-socket\") pod \"e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2\" (UID: \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\") " pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:01:50.049454 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.049391 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-istio-podinfo\") pod \"e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2\" (UID: \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\") " pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:01:50.049454 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.049451 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-istio-token\") pod \"e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2\" (UID: \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\") " pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:01:50.049747 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.049559 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-workload-socket\") pod \"e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2\" (UID: \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\") " pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:01:50.049747 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.049607 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-workload-certs\") pod \"e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2\" (UID: \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\") " pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:01:50.049747 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.049655 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-istiod-ca-cert\") pod \"e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2\" (UID: \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\") " pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:01:50.082152 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.082119 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2"] Nov 01 06:01:50.162514 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.150592 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-istio-token\") pod \"e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2\" (UID: \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\") " pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:01:50.162514 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.150692 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-workload-socket\") pod \"e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2\" (UID: \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\") " pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:01:50.162514 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.150754 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-workload-certs\") pod \"e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2\" (UID: \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\") " pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:01:50.162514 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.150794 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-istiod-ca-cert\") pod \"e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2\" (UID: \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\") " pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:01:50.162514 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.150826 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-istio-envoy\") pod \"e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2\" (UID: \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\") " pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:01:50.162514 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.150853 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-istio-data\") pod \"e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2\" (UID: \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\") " pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:01:50.162514 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.150912 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52bj2\" (UniqueName: \"kubernetes.io/projected/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-kube-api-access-52bj2\") pod \"e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2\" (UID: \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\") " pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:01:50.162514 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.150941 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-credential-socket\") pod \"e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2\" (UID: \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\") " pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:01:50.162514 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.150966 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-istio-podinfo\") pod \"e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2\" (UID: \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\") " pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:01:50.162514 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.153906 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-istio-podinfo\") pod \"e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2\" (UID: \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\") " pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:01:50.162514 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.154367 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-workload-socket\") pod \"e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2\" (UID: \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\") " pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:01:50.162514 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.154590 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-workload-certs\") pod \"e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2\" (UID: \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\") " pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:01:50.162514 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.155021 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-istiod-ca-cert\") pod \"e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2\" (UID: \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\") " pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:01:50.162514 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.161908 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-istio-data\") pod \"e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2\" (UID: \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\") " pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:01:50.162514 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.162157 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-credential-socket\") pod \"e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2\" (UID: \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\") " pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:01:50.163800 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.163751 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-istio-envoy\") pod \"e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2\" (UID: \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\") " pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:01:50.218765 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.218682 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-istio-token\") pod \"e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2\" (UID: \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\") " pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:01:50.225266 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.225201 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52bj2\" (UniqueName: \"kubernetes.io/projected/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-kube-api-access-52bj2\") pod \"e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2\" (UID: \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\") " pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:01:50.276694 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.276658 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7125-c7efa5" Nov 01 06:01:50.352444 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.352412 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec716a4b-ce8d-46c0-a9f3-864253d1de3e-kube-api-access-4xmtm\" (UniqueName: \"kubernetes.io/projected/ec716a4b-ce8d-46c0-a9f3-864253d1de3e-kube-api-access-4xmtm\") pod \"ec716a4b-ce8d-46c0-a9f3-864253d1de3e\" (UID: \"ec716a4b-ce8d-46c0-a9f3-864253d1de3e\") " Nov 01 06:01:50.354661 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.354632 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec716a4b-ce8d-46c0-a9f3-864253d1de3e-kube-api-access-4xmtm" pod "ec716a4b-ce8d-46c0-a9f3-864253d1de3e" (UID: "ec716a4b-ce8d-46c0-a9f3-864253d1de3e"). InnerVolumeSpecName "kube-api-access-4xmtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:01:50.369892 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.369859 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:01:50.453890 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.453859 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xmtm\" (UniqueName: \"kubernetes.io/projected/ec716a4b-ce8d-46c0-a9f3-864253d1de3e-kube-api-access-4xmtm\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:01:50.608763 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.608716 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a430364f-z485q"] Nov 01 06:01:50.614082 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.614058 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a430364f-z485q" Nov 01 06:01:50.665110 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.665070 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a430364f-z485q"] Nov 01 06:01:50.672071 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.672037 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:01:50.672071 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.672068 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:01:50.673050 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.673022 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:01:50.673228 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.673212 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:01:50.756985 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.756948 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpxgh\" (UniqueName: \"kubernetes.io/projected/7024010b-bb61-4b2b-ad95-fa6ca12062f0-kube-api-access-qpxgh\") pod \"maas-api-revoke-sub-a430364f-z485q\" (UID: \"7024010b-bb61-4b2b-ad95-fa6ca12062f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a430364f-z485q" Nov 01 06:01:50.757165 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.757031 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7024010b-bb61-4b2b-ad95-fa6ca12062f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a430364f-z485q\" (UID: \"7024010b-bb61-4b2b-ad95-fa6ca12062f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a430364f-z485q" Nov 01 06:01:50.757165 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.757080 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7024010b-bb61-4b2b-ad95-fa6ca12062f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a430364f-z485q\" (UID: \"7024010b-bb61-4b2b-ad95-fa6ca12062f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a430364f-z485q" Nov 01 06:01:50.800053 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.800028 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2"] Nov 01 06:01:50.800986 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:01:50.800963 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod932df59a_5bc9_4c43_a4ad_55dc8b516b7b.slice/crio-cb7c873a7ca13bf3d2603410690ed3c467630379c73b89e9ce0d9254b9552c02 WatchSource:0}: Error finding container cb7c873a7ca13bf3d2603410690ed3c467630379c73b89e9ce0d9254b9552c02: Status 404 returned error can't find the container with id cb7c873a7ca13bf3d2603410690ed3c467630379c73b89e9ce0d9254b9552c02 Nov 01 06:01:50.803517 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.803468 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:01:50.803587 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.803548 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:01:50.803587 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.803578 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:01:50.858196 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.858136 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpxgh\" (UniqueName: \"kubernetes.io/projected/7024010b-bb61-4b2b-ad95-fa6ca12062f0-kube-api-access-qpxgh\") pod \"maas-api-revoke-sub-a430364f-z485q\" (UID: \"7024010b-bb61-4b2b-ad95-fa6ca12062f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a430364f-z485q" Nov 01 06:01:50.858196 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.858200 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7024010b-bb61-4b2b-ad95-fa6ca12062f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a430364f-z485q\" (UID: \"7024010b-bb61-4b2b-ad95-fa6ca12062f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a430364f-z485q" Nov 01 06:01:50.858407 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.858226 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7024010b-bb61-4b2b-ad95-fa6ca12062f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a430364f-z485q\" (UID: \"7024010b-bb61-4b2b-ad95-fa6ca12062f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a430364f-z485q" Nov 01 06:01:50.859375 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.859318 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7024010b-bb61-4b2b-ad95-fa6ca12062f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a430364f-z485q\" (UID: \"7024010b-bb61-4b2b-ad95-fa6ca12062f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a430364f-z485q" Nov 01 06:01:50.861604 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.861447 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7024010b-bb61-4b2b-ad95-fa6ca12062f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a430364f-z485q\" (UID: \"7024010b-bb61-4b2b-ad95-fa6ca12062f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a430364f-z485q" Nov 01 06:01:50.920387 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.920355 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpxgh\" (UniqueName: \"kubernetes.io/projected/7024010b-bb61-4b2b-ad95-fa6ca12062f0-kube-api-access-qpxgh\") pod \"maas-api-revoke-sub-a430364f-z485q\" (UID: \"7024010b-bb61-4b2b-ad95-fa6ca12062f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a430364f-z485q" Nov 01 06:01:50.927346 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.927314 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a430364f-z485q" Nov 01 06:01:50.969691 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.969655 5040 generic.go:356] "Generic (PLEG): container finished" podID="ec716a4b-ce8d-46c0-a9f3-864253d1de3e" containerID="31b3ac3b17a4232718103cd93d6f75de953a6166ab58065ddbbb9e2d37ee7bfd" exitCode=137 Nov 01 06:01:50.970135 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.969731 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7125-c7efa5" Nov 01 06:01:50.970135 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.969756 5040 scope.go:122] "RemoveContainer" containerID="31b3ac3b17a4232718103cd93d6f75de953a6166ab58065ddbbb9e2d37ee7bfd" Nov 01 06:01:50.977209 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.977167 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" event={"ID":"932df59a-5bc9-4c43-a4ad-55dc8b516b7b","Type":"ContainerStarted","Data":"cc8c7cd1fb428c388d3a659f53f1391db97c46c9e8f0f3b7dd1c45f8b5dba17c"} Nov 01 06:01:50.977325 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.977219 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" event={"ID":"932df59a-5bc9-4c43-a4ad-55dc8b516b7b","Type":"ContainerStarted","Data":"cb7c873a7ca13bf3d2603410690ed3c467630379c73b89e9ce0d9254b9552c02"} Nov 01 06:01:50.986312 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.986293 5040 scope.go:122] "RemoveContainer" containerID="31b3ac3b17a4232718103cd93d6f75de953a6166ab58065ddbbb9e2d37ee7bfd" Nov 01 06:01:50.986642 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:01:50.986620 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31b3ac3b17a4232718103cd93d6f75de953a6166ab58065ddbbb9e2d37ee7bfd\": container with ID starting with 31b3ac3b17a4232718103cd93d6f75de953a6166ab58065ddbbb9e2d37ee7bfd not found: ID does not exist" containerID="31b3ac3b17a4232718103cd93d6f75de953a6166ab58065ddbbb9e2d37ee7bfd" Nov 01 06:01:50.986716 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:50.986652 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31b3ac3b17a4232718103cd93d6f75de953a6166ab58065ddbbb9e2d37ee7bfd"} err="failed to get container status \"31b3ac3b17a4232718103cd93d6f75de953a6166ab58065ddbbb9e2d37ee7bfd\": rpc error: code = NotFound desc = could not find container \"31b3ac3b17a4232718103cd93d6f75de953a6166ab58065ddbbb9e2d37ee7bfd\": container with ID starting with 31b3ac3b17a4232718103cd93d6f75de953a6166ab58065ddbbb9e2d37ee7bfd not found: ID does not exist" Nov 01 06:01:51.101709 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:51.101656 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" podStartSLOduration=2.101640775 podStartE2EDuration="2.101640775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:01:51.099999743 +0000 UTC m=+989.755116911" watchObservedRunningTime="2026-11-01 06:01:51.101640775 +0000 UTC m=+989.756757939" Nov 01 06:01:51.152448 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:01:51.152360 5040 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7125-c7efa5\" is forbidden: User \"system:node:ip-10-0-143-195.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-195.us-west-1.compute.internal' and this object" podUID="ec716a4b-ce8d-46c0-a9f3-864253d1de3e" pod="opendatahub/test-curl-7125-c7efa5" Nov 01 06:01:51.315781 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:51.315742 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a430364f-z485q"] Nov 01 06:01:51.317081 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:01:51.317054 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7024010b_bb61_4b2b_ad95_fa6ca12062f0.slice/crio-14befc5d12198a035be811e7564c56926dc59ba060f3830f514193242e484dd3 WatchSource:0}: Error finding container 14befc5d12198a035be811e7564c56926dc59ba060f3830f514193242e484dd3: Status 404 returned error can't find the container with id 14befc5d12198a035be811e7564c56926dc59ba060f3830f514193242e484dd3 Nov 01 06:01:51.369992 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:51.369964 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:01:51.375288 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:51.375263 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:01:51.427469 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:01:51.427376 5040 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7125-c7efa5\" is forbidden: User \"system:node:ip-10-0-143-195.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-195.us-west-1.compute.internal' and this object" podUID="ec716a4b-ce8d-46c0-a9f3-864253d1de3e" pod="opendatahub/test-curl-7125-c7efa5" Nov 01 06:01:51.515241 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:51.515209 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec716a4b-ce8d-46c0-a9f3-864253d1de3e" path="/var/lib/kubelet/pods/ec716a4b-ce8d-46c0-a9f3-864253d1de3e/volumes" Nov 01 06:01:51.986641 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:51.986598 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a430364f-z485q" event={"ID":"7024010b-bb61-4b2b-ad95-fa6ca12062f0","Type":"ContainerStarted","Data":"c8314eb0927c9fb4effcc5ef444f95fbf819962f5615e060c17cf9b86acfb286"} Nov 01 06:01:51.987142 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:51.986656 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a430364f-z485q" event={"ID":"7024010b-bb61-4b2b-ad95-fa6ca12062f0","Type":"ContainerStarted","Data":"14befc5d12198a035be811e7564c56926dc59ba060f3830f514193242e484dd3"} Nov 01 06:01:51.987142 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:51.987091 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:01:51.987927 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:51.987907 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:01:52.106878 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:52.106834 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a430364f-z485q" podStartSLOduration=2.106819384 podStartE2EDuration="2.106819384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:01:52.103926643 +0000 UTC m=+990.759043814" watchObservedRunningTime="2026-11-01 06:01:52.106819384 +0000 UTC m=+990.761936550" Nov 01 06:01:52.994767 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:52.994730 5040 generic.go:356] "Generic (PLEG): container finished" podID="7024010b-bb61-4b2b-ad95-fa6ca12062f0" containerID="c8314eb0927c9fb4effcc5ef444f95fbf819962f5615e060c17cf9b86acfb286" exitCode=0 Nov 01 06:01:52.995248 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:52.994801 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a430364f-z485q" event={"ID":"7024010b-bb61-4b2b-ad95-fa6ca12062f0","Type":"ContainerDied","Data":"c8314eb0927c9fb4effcc5ef444f95fbf819962f5615e060c17cf9b86acfb286"} Nov 01 06:01:54.184325 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:54.184296 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a430364f-z485q" Nov 01 06:01:54.297754 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:54.297670 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7024010b-bb61-4b2b-ad95-fa6ca12062f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7024010b-bb61-4b2b-ad95-fa6ca12062f0-maas-api-openshift-ca\") pod \"7024010b-bb61-4b2b-ad95-fa6ca12062f0\" (UID: \"7024010b-bb61-4b2b-ad95-fa6ca12062f0\") " Nov 01 06:01:54.297931 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:54.297793 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7024010b-bb61-4b2b-ad95-fa6ca12062f0-kube-api-access-qpxgh\" (UniqueName: \"kubernetes.io/projected/7024010b-bb61-4b2b-ad95-fa6ca12062f0-kube-api-access-qpxgh\") pod \"7024010b-bb61-4b2b-ad95-fa6ca12062f0\" (UID: \"7024010b-bb61-4b2b-ad95-fa6ca12062f0\") " Nov 01 06:01:54.297931 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:54.297867 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7024010b-bb61-4b2b-ad95-fa6ca12062f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7024010b-bb61-4b2b-ad95-fa6ca12062f0-maas-api-xks-ca\") pod \"7024010b-bb61-4b2b-ad95-fa6ca12062f0\" (UID: \"7024010b-bb61-4b2b-ad95-fa6ca12062f0\") " Nov 01 06:01:54.298139 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:54.298101 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7024010b-bb61-4b2b-ad95-fa6ca12062f0-maas-api-openshift-ca" pod "7024010b-bb61-4b2b-ad95-fa6ca12062f0" (UID: "7024010b-bb61-4b2b-ad95-fa6ca12062f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:01:54.299989 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:54.299961 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7024010b-bb61-4b2b-ad95-fa6ca12062f0-maas-api-xks-ca" pod "7024010b-bb61-4b2b-ad95-fa6ca12062f0" (UID: "7024010b-bb61-4b2b-ad95-fa6ca12062f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:01:54.300085 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:54.299973 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7024010b-bb61-4b2b-ad95-fa6ca12062f0-kube-api-access-qpxgh" pod "7024010b-bb61-4b2b-ad95-fa6ca12062f0" (UID: "7024010b-bb61-4b2b-ad95-fa6ca12062f0"). InnerVolumeSpecName "kube-api-access-qpxgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:01:54.399907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:54.399874 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpxgh\" (UniqueName: \"kubernetes.io/projected/7024010b-bb61-4b2b-ad95-fa6ca12062f0-kube-api-access-qpxgh\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:01:54.399907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:54.399905 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7024010b-bb61-4b2b-ad95-fa6ca12062f0-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:01:54.399907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:54.399915 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7024010b-bb61-4b2b-ad95-fa6ca12062f0-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:01:55.007010 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:55.006980 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a430364f-z485q" Nov 01 06:01:55.007010 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:55.006993 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a430364f-z485q" event={"ID":"7024010b-bb61-4b2b-ad95-fa6ca12062f0","Type":"ContainerDied","Data":"14befc5d12198a035be811e7564c56926dc59ba060f3830f514193242e484dd3"} Nov 01 06:01:55.007227 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:01:55.007043 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14befc5d12198a035be811e7564c56926dc59ba060f3830f514193242e484dd3" Nov 01 06:02:09.985462 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:02:09.985420 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:02:16.837081 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:16.837041 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c08529d-wsqpq"] Nov 01 06:02:16.837494 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:16.837466 5040 cpu_manager.go:414] "containerMap: removing container" podUID="7024010b-bb61-4b2b-ad95-fa6ca12062f0" containerName="revoke-keys" Nov 01 06:02:16.837547 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:16.837496 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7024010b-bb61-4b2b-ad95-fa6ca12062f0" containerName="revoke-keys" Nov 01 06:02:16.837621 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:16.837612 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="7024010b-bb61-4b2b-ad95-fa6ca12062f0" containerName="revoke-keys" Nov 01 06:02:16.840580 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:16.840563 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c08529d-wsqpq" Nov 01 06:02:16.945712 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:16.945673 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:02:16.945712 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:16.945698 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:02:16.945712 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:16.945673 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:02:16.945978 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:16.945773 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:02:16.947218 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:16.947197 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/948a4c8b-5e28-4fe0-8933-4f0ac460002e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c08529d-wsqpq\" (UID: \"948a4c8b-5e28-4fe0-8933-4f0ac460002e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c08529d-wsqpq" Nov 01 06:02:16.947271 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:16.947243 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/948a4c8b-5e28-4fe0-8933-4f0ac460002e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c08529d-wsqpq\" (UID: \"948a4c8b-5e28-4fe0-8933-4f0ac460002e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c08529d-wsqpq" Nov 01 06:02:16.947386 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:16.947374 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k4ch\" (UniqueName: \"kubernetes.io/projected/948a4c8b-5e28-4fe0-8933-4f0ac460002e-kube-api-access-7k4ch\") pod \"maas-api-revoke-sub-7c08529d-wsqpq\" (UID: \"948a4c8b-5e28-4fe0-8933-4f0ac460002e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c08529d-wsqpq" Nov 01 06:02:16.948558 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:16.948534 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c08529d-wsqpq"] Nov 01 06:02:17.047857 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:17.047818 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/948a4c8b-5e28-4fe0-8933-4f0ac460002e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c08529d-wsqpq\" (UID: \"948a4c8b-5e28-4fe0-8933-4f0ac460002e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c08529d-wsqpq" Nov 01 06:02:17.048058 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:17.047888 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/948a4c8b-5e28-4fe0-8933-4f0ac460002e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c08529d-wsqpq\" (UID: \"948a4c8b-5e28-4fe0-8933-4f0ac460002e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c08529d-wsqpq" Nov 01 06:02:17.048058 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:17.048028 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k4ch\" (UniqueName: \"kubernetes.io/projected/948a4c8b-5e28-4fe0-8933-4f0ac460002e-kube-api-access-7k4ch\") pod \"maas-api-revoke-sub-7c08529d-wsqpq\" (UID: \"948a4c8b-5e28-4fe0-8933-4f0ac460002e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c08529d-wsqpq" Nov 01 06:02:17.048552 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:17.048496 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/948a4c8b-5e28-4fe0-8933-4f0ac460002e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c08529d-wsqpq\" (UID: \"948a4c8b-5e28-4fe0-8933-4f0ac460002e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c08529d-wsqpq" Nov 01 06:02:17.050218 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:17.050195 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/948a4c8b-5e28-4fe0-8933-4f0ac460002e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c08529d-wsqpq\" (UID: \"948a4c8b-5e28-4fe0-8933-4f0ac460002e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c08529d-wsqpq" Nov 01 06:02:17.109913 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:17.109833 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k4ch\" (UniqueName: \"kubernetes.io/projected/948a4c8b-5e28-4fe0-8933-4f0ac460002e-kube-api-access-7k4ch\") pod \"maas-api-revoke-sub-7c08529d-wsqpq\" (UID: \"948a4c8b-5e28-4fe0-8933-4f0ac460002e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c08529d-wsqpq" Nov 01 06:02:17.154336 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:17.154302 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c08529d-wsqpq" Nov 01 06:02:17.530590 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:17.530559 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c08529d-wsqpq"] Nov 01 06:02:17.531700 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:02:17.531675 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod948a4c8b_5e28_4fe0_8933_4f0ac460002e.slice/crio-07cd7a6e49525a324fac2a7be4c18f0f2f389d73ad9b760bae84f624bf0a83a3 WatchSource:0}: Error finding container 07cd7a6e49525a324fac2a7be4c18f0f2f389d73ad9b760bae84f624bf0a83a3: Status 404 returned error can't find the container with id 07cd7a6e49525a324fac2a7be4c18f0f2f389d73ad9b760bae84f624bf0a83a3 Nov 01 06:02:18.143605 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:18.143516 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c08529d-wsqpq" event={"ID":"948a4c8b-5e28-4fe0-8933-4f0ac460002e","Type":"ContainerStarted","Data":"f378d9633cc8088e84c2bf64e9b9841433a504ebbe5a4481f74d5fb440d2e121"} Nov 01 06:02:18.143605 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:18.143600 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c08529d-wsqpq" event={"ID":"948a4c8b-5e28-4fe0-8933-4f0ac460002e","Type":"ContainerStarted","Data":"07cd7a6e49525a324fac2a7be4c18f0f2f389d73ad9b760bae84f624bf0a83a3"} Nov 01 06:02:18.262228 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:18.262176 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c08529d-wsqpq" podStartSLOduration=2.262163853 podStartE2EDuration="2.262163853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:02:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:02:18.261829724 +0000 UTC m=+1016.916946891" watchObservedRunningTime="2026-11-01 06:02:18.262163853 +0000 UTC m=+1016.917281019" Nov 01 06:02:19.162940 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:19.162904 5040 generic.go:356] "Generic (PLEG): container finished" podID="948a4c8b-5e28-4fe0-8933-4f0ac460002e" containerID="f378d9633cc8088e84c2bf64e9b9841433a504ebbe5a4481f74d5fb440d2e121" exitCode=0 Nov 01 06:02:19.163459 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:19.163098 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c08529d-wsqpq" event={"ID":"948a4c8b-5e28-4fe0-8933-4f0ac460002e","Type":"ContainerDied","Data":"f378d9633cc8088e84c2bf64e9b9841433a504ebbe5a4481f74d5fb440d2e121"} Nov 01 06:02:20.311870 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:20.311845 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c08529d-wsqpq" Nov 01 06:02:20.382424 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:20.382389 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/948a4c8b-5e28-4fe0-8933-4f0ac460002e-kube-api-access-7k4ch\" (UniqueName: \"kubernetes.io/projected/948a4c8b-5e28-4fe0-8933-4f0ac460002e-kube-api-access-7k4ch\") pod \"948a4c8b-5e28-4fe0-8933-4f0ac460002e\" (UID: \"948a4c8b-5e28-4fe0-8933-4f0ac460002e\") " Nov 01 06:02:20.382636 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:20.382473 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/948a4c8b-5e28-4fe0-8933-4f0ac460002e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/948a4c8b-5e28-4fe0-8933-4f0ac460002e-maas-api-openshift-ca\") pod \"948a4c8b-5e28-4fe0-8933-4f0ac460002e\" (UID: \"948a4c8b-5e28-4fe0-8933-4f0ac460002e\") " Nov 01 06:02:20.382636 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:20.382557 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/948a4c8b-5e28-4fe0-8933-4f0ac460002e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/948a4c8b-5e28-4fe0-8933-4f0ac460002e-maas-api-xks-ca\") pod \"948a4c8b-5e28-4fe0-8933-4f0ac460002e\" (UID: \"948a4c8b-5e28-4fe0-8933-4f0ac460002e\") " Nov 01 06:02:20.382860 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:20.382827 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/948a4c8b-5e28-4fe0-8933-4f0ac460002e-maas-api-openshift-ca" pod "948a4c8b-5e28-4fe0-8933-4f0ac460002e" (UID: "948a4c8b-5e28-4fe0-8933-4f0ac460002e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:02:20.383000 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:20.382947 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/948a4c8b-5e28-4fe0-8933-4f0ac460002e-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:02:20.384680 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:20.384658 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/948a4c8b-5e28-4fe0-8933-4f0ac460002e-kube-api-access-7k4ch" pod "948a4c8b-5e28-4fe0-8933-4f0ac460002e" (UID: "948a4c8b-5e28-4fe0-8933-4f0ac460002e"). InnerVolumeSpecName "kube-api-access-7k4ch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:02:20.385023 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:20.385003 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/948a4c8b-5e28-4fe0-8933-4f0ac460002e-maas-api-xks-ca" pod "948a4c8b-5e28-4fe0-8933-4f0ac460002e" (UID: "948a4c8b-5e28-4fe0-8933-4f0ac460002e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:02:20.484259 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:20.484173 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/948a4c8b-5e28-4fe0-8933-4f0ac460002e-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:02:20.484259 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:20.484209 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k4ch\" (UniqueName: \"kubernetes.io/projected/948a4c8b-5e28-4fe0-8933-4f0ac460002e-kube-api-access-7k4ch\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:02:21.175626 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:21.175591 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c08529d-wsqpq" Nov 01 06:02:21.175799 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:21.175593 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c08529d-wsqpq" event={"ID":"948a4c8b-5e28-4fe0-8933-4f0ac460002e","Type":"ContainerDied","Data":"07cd7a6e49525a324fac2a7be4c18f0f2f389d73ad9b760bae84f624bf0a83a3"} Nov 01 06:02:21.175799 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:21.175716 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07cd7a6e49525a324fac2a7be4c18f0f2f389d73ad9b760bae84f624bf0a83a3" Nov 01 06:02:33.917263 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:33.917185 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1f30d57-nz2b6"] Nov 01 06:02:33.917758 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:33.917647 5040 cpu_manager.go:414] "containerMap: removing container" podUID="948a4c8b-5e28-4fe0-8933-4f0ac460002e" containerName="revoke-keys" Nov 01 06:02:33.917758 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:33.917660 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="948a4c8b-5e28-4fe0-8933-4f0ac460002e" containerName="revoke-keys" Nov 01 06:02:33.917758 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:33.917737 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="948a4c8b-5e28-4fe0-8933-4f0ac460002e" containerName="revoke-keys" Nov 01 06:02:33.920862 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:33.920846 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f30d57-nz2b6" Nov 01 06:02:33.975233 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:33.975199 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:02:33.975430 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:33.975280 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:02:33.975430 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:33.975391 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:02:34.025704 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:34.025667 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:02:34.028385 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:34.028357 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1f30d57-nz2b6"] Nov 01 06:02:34.028774 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:34.028752 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cw7h\" (UniqueName: \"kubernetes.io/projected/82f07425-72e6-43f1-9f54-fb18aef7a7a6-kube-api-access-7cw7h\") pod \"maas-api-revoke-sub-b1f30d57-nz2b6\" (UID: \"82f07425-72e6-43f1-9f54-fb18aef7a7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f30d57-nz2b6" Nov 01 06:02:34.028872 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:34.028842 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f07425-72e6-43f1-9f54-fb18aef7a7a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1f30d57-nz2b6\" (UID: \"82f07425-72e6-43f1-9f54-fb18aef7a7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f30d57-nz2b6" Nov 01 06:02:34.028958 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:34.028943 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f07425-72e6-43f1-9f54-fb18aef7a7a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1f30d57-nz2b6\" (UID: \"82f07425-72e6-43f1-9f54-fb18aef7a7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f30d57-nz2b6" Nov 01 06:02:34.129603 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:34.129570 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cw7h\" (UniqueName: \"kubernetes.io/projected/82f07425-72e6-43f1-9f54-fb18aef7a7a6-kube-api-access-7cw7h\") pod \"maas-api-revoke-sub-b1f30d57-nz2b6\" (UID: \"82f07425-72e6-43f1-9f54-fb18aef7a7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f30d57-nz2b6" Nov 01 06:02:34.129760 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:34.129617 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f07425-72e6-43f1-9f54-fb18aef7a7a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1f30d57-nz2b6\" (UID: \"82f07425-72e6-43f1-9f54-fb18aef7a7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f30d57-nz2b6" Nov 01 06:02:34.129760 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:34.129698 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f07425-72e6-43f1-9f54-fb18aef7a7a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1f30d57-nz2b6\" (UID: \"82f07425-72e6-43f1-9f54-fb18aef7a7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f30d57-nz2b6" Nov 01 06:02:34.130298 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:34.130272 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f07425-72e6-43f1-9f54-fb18aef7a7a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1f30d57-nz2b6\" (UID: \"82f07425-72e6-43f1-9f54-fb18aef7a7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f30d57-nz2b6" Nov 01 06:02:34.132043 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:34.132020 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f07425-72e6-43f1-9f54-fb18aef7a7a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1f30d57-nz2b6\" (UID: \"82f07425-72e6-43f1-9f54-fb18aef7a7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f30d57-nz2b6" Nov 01 06:02:34.191304 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:34.191235 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cw7h\" (UniqueName: \"kubernetes.io/projected/82f07425-72e6-43f1-9f54-fb18aef7a7a6-kube-api-access-7cw7h\") pod \"maas-api-revoke-sub-b1f30d57-nz2b6\" (UID: \"82f07425-72e6-43f1-9f54-fb18aef7a7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f30d57-nz2b6" Nov 01 06:02:34.233997 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:34.233971 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f30d57-nz2b6" Nov 01 06:02:34.606693 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:34.606660 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1f30d57-nz2b6"] Nov 01 06:02:34.608043 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:02:34.608014 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f07425_72e6_43f1_9f54_fb18aef7a7a6.slice/crio-6ff08a2cc127136c9792f82ee9210844302a05cc0d3422d6225152d6f25f11a9 WatchSource:0}: Error finding container 6ff08a2cc127136c9792f82ee9210844302a05cc0d3422d6225152d6f25f11a9: Status 404 returned error can't find the container with id 6ff08a2cc127136c9792f82ee9210844302a05cc0d3422d6225152d6f25f11a9 Nov 01 06:02:35.253954 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:35.253916 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f30d57-nz2b6" event={"ID":"82f07425-72e6-43f1-9f54-fb18aef7a7a6","Type":"ContainerStarted","Data":"96bf31c5422b4ac116d768e29eb07f2213558a2461be29ad870d9a04312e9e47"} Nov 01 06:02:35.254408 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:35.253970 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f30d57-nz2b6" event={"ID":"82f07425-72e6-43f1-9f54-fb18aef7a7a6","Type":"ContainerStarted","Data":"6ff08a2cc127136c9792f82ee9210844302a05cc0d3422d6225152d6f25f11a9"} Nov 01 06:02:35.374556 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:35.374471 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f30d57-nz2b6" podStartSLOduration=2.374453952 podStartE2EDuration="2.374453952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:02:35.370878921 +0000 UTC m=+1034.025996093" watchObservedRunningTime="2026-11-01 06:02:35.374453952 +0000 UTC m=+1034.029571120" Nov 01 06:02:36.263957 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:36.263921 5040 generic.go:356] "Generic (PLEG): container finished" podID="82f07425-72e6-43f1-9f54-fb18aef7a7a6" containerID="96bf31c5422b4ac116d768e29eb07f2213558a2461be29ad870d9a04312e9e47" exitCode=0 Nov 01 06:02:36.264511 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:36.264053 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f30d57-nz2b6" event={"ID":"82f07425-72e6-43f1-9f54-fb18aef7a7a6","Type":"ContainerDied","Data":"96bf31c5422b4ac116d768e29eb07f2213558a2461be29ad870d9a04312e9e47"} Nov 01 06:02:37.460778 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:37.460751 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f30d57-nz2b6" Nov 01 06:02:37.570076 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:37.569971 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82f07425-72e6-43f1-9f54-fb18aef7a7a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f07425-72e6-43f1-9f54-fb18aef7a7a6-maas-api-openshift-ca\") pod \"82f07425-72e6-43f1-9f54-fb18aef7a7a6\" (UID: \"82f07425-72e6-43f1-9f54-fb18aef7a7a6\") " Nov 01 06:02:37.570076 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:37.570032 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82f07425-72e6-43f1-9f54-fb18aef7a7a6-kube-api-access-7cw7h\" (UniqueName: \"kubernetes.io/projected/82f07425-72e6-43f1-9f54-fb18aef7a7a6-kube-api-access-7cw7h\") pod \"82f07425-72e6-43f1-9f54-fb18aef7a7a6\" (UID: \"82f07425-72e6-43f1-9f54-fb18aef7a7a6\") " Nov 01 06:02:37.570316 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:37.570080 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82f07425-72e6-43f1-9f54-fb18aef7a7a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f07425-72e6-43f1-9f54-fb18aef7a7a6-maas-api-xks-ca\") pod \"82f07425-72e6-43f1-9f54-fb18aef7a7a6\" (UID: \"82f07425-72e6-43f1-9f54-fb18aef7a7a6\") " Nov 01 06:02:37.570381 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:37.570315 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82f07425-72e6-43f1-9f54-fb18aef7a7a6-maas-api-openshift-ca" pod "82f07425-72e6-43f1-9f54-fb18aef7a7a6" (UID: "82f07425-72e6-43f1-9f54-fb18aef7a7a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:02:37.570470 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:37.570452 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f07425-72e6-43f1-9f54-fb18aef7a7a6-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:02:37.572274 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:37.572235 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f07425-72e6-43f1-9f54-fb18aef7a7a6-kube-api-access-7cw7h" pod "82f07425-72e6-43f1-9f54-fb18aef7a7a6" (UID: "82f07425-72e6-43f1-9f54-fb18aef7a7a6"). InnerVolumeSpecName "kube-api-access-7cw7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:02:37.572274 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:37.572241 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82f07425-72e6-43f1-9f54-fb18aef7a7a6-maas-api-xks-ca" pod "82f07425-72e6-43f1-9f54-fb18aef7a7a6" (UID: "82f07425-72e6-43f1-9f54-fb18aef7a7a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:02:37.670975 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:37.670940 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cw7h\" (UniqueName: \"kubernetes.io/projected/82f07425-72e6-43f1-9f54-fb18aef7a7a6-kube-api-access-7cw7h\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:02:37.670975 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:37.670969 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f07425-72e6-43f1-9f54-fb18aef7a7a6-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:02:38.275666 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:38.275631 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f30d57-nz2b6" Nov 01 06:02:38.275841 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:38.275672 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f30d57-nz2b6" event={"ID":"82f07425-72e6-43f1-9f54-fb18aef7a7a6","Type":"ContainerDied","Data":"6ff08a2cc127136c9792f82ee9210844302a05cc0d3422d6225152d6f25f11a9"} Nov 01 06:02:38.275841 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:38.275716 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ff08a2cc127136c9792f82ee9210844302a05cc0d3422d6225152d6f25f11a9" Nov 01 06:02:39.419340 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:39.419287 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfcfb1b2-qxgrb"] Nov 01 06:02:39.420059 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:39.420035 5040 cpu_manager.go:414] "containerMap: removing container" podUID="82f07425-72e6-43f1-9f54-fb18aef7a7a6" containerName="revoke-keys" Nov 01 06:02:39.420255 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:39.420236 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82f07425-72e6-43f1-9f54-fb18aef7a7a6" containerName="revoke-keys" Nov 01 06:02:39.420552 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:39.420532 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="82f07425-72e6-43f1-9f54-fb18aef7a7a6" containerName="revoke-keys" Nov 01 06:02:39.424885 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:39.424861 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcfb1b2-qxgrb" Nov 01 06:02:39.476058 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:39.476028 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfcfb1b2-qxgrb"] Nov 01 06:02:39.491352 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:39.491319 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/311f01ab-e4e0-4661-bd54-256ff29f3d39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfcfb1b2-qxgrb\" (UID: \"311f01ab-e4e0-4661-bd54-256ff29f3d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcfb1b2-qxgrb" Nov 01 06:02:39.491559 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:39.491363 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzx6t\" (UniqueName: \"kubernetes.io/projected/311f01ab-e4e0-4661-bd54-256ff29f3d39-kube-api-access-tzx6t\") pod \"maas-api-revoke-sub-dfcfb1b2-qxgrb\" (UID: \"311f01ab-e4e0-4661-bd54-256ff29f3d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcfb1b2-qxgrb" Nov 01 06:02:39.491631 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:39.491560 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/311f01ab-e4e0-4661-bd54-256ff29f3d39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfcfb1b2-qxgrb\" (UID: \"311f01ab-e4e0-4661-bd54-256ff29f3d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcfb1b2-qxgrb" Nov 01 06:02:39.523702 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:39.523492 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:02:39.523702 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:39.523491 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:02:39.523702 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:39.523615 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:02:39.528307 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:39.528283 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:02:39.593013 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:39.592986 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/311f01ab-e4e0-4661-bd54-256ff29f3d39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfcfb1b2-qxgrb\" (UID: \"311f01ab-e4e0-4661-bd54-256ff29f3d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcfb1b2-qxgrb" Nov 01 06:02:39.593169 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:39.593042 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/311f01ab-e4e0-4661-bd54-256ff29f3d39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfcfb1b2-qxgrb\" (UID: \"311f01ab-e4e0-4661-bd54-256ff29f3d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcfb1b2-qxgrb" Nov 01 06:02:39.593169 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:39.593068 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzx6t\" (UniqueName: \"kubernetes.io/projected/311f01ab-e4e0-4661-bd54-256ff29f3d39-kube-api-access-tzx6t\") pod \"maas-api-revoke-sub-dfcfb1b2-qxgrb\" (UID: \"311f01ab-e4e0-4661-bd54-256ff29f3d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcfb1b2-qxgrb" Nov 01 06:02:39.612499 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:39.593686 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/311f01ab-e4e0-4661-bd54-256ff29f3d39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfcfb1b2-qxgrb\" (UID: \"311f01ab-e4e0-4661-bd54-256ff29f3d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcfb1b2-qxgrb" Nov 01 06:02:39.620200 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:39.619032 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/311f01ab-e4e0-4661-bd54-256ff29f3d39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfcfb1b2-qxgrb\" (UID: \"311f01ab-e4e0-4661-bd54-256ff29f3d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcfb1b2-qxgrb" Nov 01 06:02:39.654565 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:39.654536 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzx6t\" (UniqueName: \"kubernetes.io/projected/311f01ab-e4e0-4661-bd54-256ff29f3d39-kube-api-access-tzx6t\") pod \"maas-api-revoke-sub-dfcfb1b2-qxgrb\" (UID: \"311f01ab-e4e0-4661-bd54-256ff29f3d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcfb1b2-qxgrb" Nov 01 06:02:39.737983 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:39.737895 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcfb1b2-qxgrb" Nov 01 06:02:39.972826 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:02:39.972793 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:02:40.123416 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:40.123373 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfcfb1b2-qxgrb"] Nov 01 06:02:40.292672 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:40.292634 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcfb1b2-qxgrb" event={"ID":"311f01ab-e4e0-4661-bd54-256ff29f3d39","Type":"ContainerStarted","Data":"bbbaf40225a72ef8049fa6ad95b90eb22f32a01e70bb90b0130dc4a143d9cbbd"} Nov 01 06:02:41.322128 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:41.322092 5040 generic.go:356] "Generic (PLEG): container finished" podID="311f01ab-e4e0-4661-bd54-256ff29f3d39" containerID="a84e8aabdca1f19d664eebaabb569e921f2f97d61011f7bba2ad549f4071b295" exitCode=0 Nov 01 06:02:41.322569 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:41.322172 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcfb1b2-qxgrb" event={"ID":"311f01ab-e4e0-4661-bd54-256ff29f3d39","Type":"ContainerDied","Data":"a84e8aabdca1f19d664eebaabb569e921f2f97d61011f7bba2ad549f4071b295"} Nov 01 06:02:42.495142 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:42.495113 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcfb1b2-qxgrb" Nov 01 06:02:42.643679 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:42.643593 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/311f01ab-e4e0-4661-bd54-256ff29f3d39-kube-api-access-tzx6t\" (UniqueName: \"kubernetes.io/projected/311f01ab-e4e0-4661-bd54-256ff29f3d39-kube-api-access-tzx6t\") pod \"311f01ab-e4e0-4661-bd54-256ff29f3d39\" (UID: \"311f01ab-e4e0-4661-bd54-256ff29f3d39\") " Nov 01 06:02:42.643679 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:42.643676 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/311f01ab-e4e0-4661-bd54-256ff29f3d39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/311f01ab-e4e0-4661-bd54-256ff29f3d39-maas-api-openshift-ca\") pod \"311f01ab-e4e0-4661-bd54-256ff29f3d39\" (UID: \"311f01ab-e4e0-4661-bd54-256ff29f3d39\") " Nov 01 06:02:42.643901 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:42.643726 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/311f01ab-e4e0-4661-bd54-256ff29f3d39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/311f01ab-e4e0-4661-bd54-256ff29f3d39-maas-api-xks-ca\") pod \"311f01ab-e4e0-4661-bd54-256ff29f3d39\" (UID: \"311f01ab-e4e0-4661-bd54-256ff29f3d39\") " Nov 01 06:02:42.644541 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:42.644505 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311f01ab-e4e0-4661-bd54-256ff29f3d39-maas-api-openshift-ca" pod "311f01ab-e4e0-4661-bd54-256ff29f3d39" (UID: "311f01ab-e4e0-4661-bd54-256ff29f3d39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:02:42.646053 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:42.646027 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311f01ab-e4e0-4661-bd54-256ff29f3d39-kube-api-access-tzx6t" pod "311f01ab-e4e0-4661-bd54-256ff29f3d39" (UID: "311f01ab-e4e0-4661-bd54-256ff29f3d39"). InnerVolumeSpecName "kube-api-access-tzx6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:02:42.646411 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:42.646385 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/311f01ab-e4e0-4661-bd54-256ff29f3d39-maas-api-xks-ca" pod "311f01ab-e4e0-4661-bd54-256ff29f3d39" (UID: "311f01ab-e4e0-4661-bd54-256ff29f3d39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:02:42.744805 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:42.744754 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/311f01ab-e4e0-4661-bd54-256ff29f3d39-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:02:42.744805 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:42.744791 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/311f01ab-e4e0-4661-bd54-256ff29f3d39-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:02:42.744805 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:42.744806 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzx6t\" (UniqueName: \"kubernetes.io/projected/311f01ab-e4e0-4661-bd54-256ff29f3d39-kube-api-access-tzx6t\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:02:43.336044 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:43.336013 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcfb1b2-qxgrb" Nov 01 06:02:43.336248 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:43.336054 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcfb1b2-qxgrb" event={"ID":"311f01ab-e4e0-4661-bd54-256ff29f3d39","Type":"ContainerDied","Data":"bbbaf40225a72ef8049fa6ad95b90eb22f32a01e70bb90b0130dc4a143d9cbbd"} Nov 01 06:02:43.336248 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:43.336102 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbbaf40225a72ef8049fa6ad95b90eb22f32a01e70bb90b0130dc4a143d9cbbd" Nov 01 06:02:44.677382 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:44.677347 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bddf4e87-zcpb9"] Nov 01 06:02:44.677916 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:44.677769 5040 cpu_manager.go:414] "containerMap: removing container" podUID="311f01ab-e4e0-4661-bd54-256ff29f3d39" containerName="revoke-keys" Nov 01 06:02:44.677916 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:44.677781 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="311f01ab-e4e0-4661-bd54-256ff29f3d39" containerName="revoke-keys" Nov 01 06:02:44.677916 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:44.677873 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="311f01ab-e4e0-4661-bd54-256ff29f3d39" containerName="revoke-keys" Nov 01 06:02:44.681066 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:44.681046 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddf4e87-zcpb9" Nov 01 06:02:44.735389 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:44.735338 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:02:44.735561 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:44.735401 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:02:44.785660 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:44.785630 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:02:44.785850 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:44.785773 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:02:44.790337 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:44.790307 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bddf4e87-zcpb9"] Nov 01 06:02:44.865975 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:44.865941 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bddf4e87-zcpb9\" (UID: \"0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddf4e87-zcpb9" Nov 01 06:02:44.868286 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:44.866189 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bddf4e87-zcpb9\" (UID: \"0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddf4e87-zcpb9" Nov 01 06:02:44.868286 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:44.866231 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f82z8\" (UniqueName: \"kubernetes.io/projected/0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5-kube-api-access-f82z8\") pod \"maas-api-revoke-sub-bddf4e87-zcpb9\" (UID: \"0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddf4e87-zcpb9" Nov 01 06:02:44.967044 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:44.966967 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bddf4e87-zcpb9\" (UID: \"0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddf4e87-zcpb9" Nov 01 06:02:44.967044 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:44.967012 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bddf4e87-zcpb9\" (UID: \"0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddf4e87-zcpb9" Nov 01 06:02:44.967044 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:44.967030 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f82z8\" (UniqueName: \"kubernetes.io/projected/0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5-kube-api-access-f82z8\") pod \"maas-api-revoke-sub-bddf4e87-zcpb9\" (UID: \"0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddf4e87-zcpb9" Nov 01 06:02:44.967777 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:44.967755 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bddf4e87-zcpb9\" (UID: \"0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddf4e87-zcpb9" Nov 01 06:02:44.969318 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:44.969300 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bddf4e87-zcpb9\" (UID: \"0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddf4e87-zcpb9" Nov 01 06:02:45.029548 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:45.029514 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f82z8\" (UniqueName: \"kubernetes.io/projected/0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5-kube-api-access-f82z8\") pod \"maas-api-revoke-sub-bddf4e87-zcpb9\" (UID: \"0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddf4e87-zcpb9" Nov 01 06:02:45.295332 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:45.295297 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddf4e87-zcpb9" Nov 01 06:02:45.352417 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:45.352369 5040 generic.go:356] "Generic (PLEG): container finished" podID="932df59a-5bc9-4c43-a4ad-55dc8b516b7b" containerID="cc8c7cd1fb428c388d3a659f53f1391db97c46c9e8f0f3b7dd1c45f8b5dba17c" exitCode=0 Nov 01 06:02:45.362640 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:45.352449 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" event={"ID":"932df59a-5bc9-4c43-a4ad-55dc8b516b7b","Type":"ContainerDied","Data":"cc8c7cd1fb428c388d3a659f53f1391db97c46c9e8f0f3b7dd1c45f8b5dba17c"} Nov 01 06:02:45.362640 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:45.353267 5040 scope.go:122] "RemoveContainer" containerID="cc8c7cd1fb428c388d3a659f53f1391db97c46c9e8f0f3b7dd1c45f8b5dba17c" Nov 01 06:02:45.362640 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:45.357045 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:02:45.362640 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:45.357114 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:02:45.362640 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:45.357143 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:02:45.668584 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:45.668560 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bddf4e87-zcpb9"] Nov 01 06:02:45.669657 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:02:45.669630 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b9e2bbb_1d47_4cc4_bb0c_f50e6795e9e5.slice/crio-b8cbbca9bbf7d2798974e9d5fb256d5f54a52f2e8b13563b05859560ab354926 WatchSource:0}: Error finding container b8cbbca9bbf7d2798974e9d5fb256d5f54a52f2e8b13563b05859560ab354926: Status 404 returned error can't find the container with id b8cbbca9bbf7d2798974e9d5fb256d5f54a52f2e8b13563b05859560ab354926 Nov 01 06:02:46.359655 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:46.359620 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddf4e87-zcpb9" event={"ID":"0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5","Type":"ContainerStarted","Data":"02038dfa70f16ef0855d455cfd5d3a85a2dfa62284139c2cc7846cb58d2cf9dc"} Nov 01 06:02:46.360098 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:46.359683 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddf4e87-zcpb9" event={"ID":"0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5","Type":"ContainerStarted","Data":"b8cbbca9bbf7d2798974e9d5fb256d5f54a52f2e8b13563b05859560ab354926"} Nov 01 06:02:46.361901 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:46.361877 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" event={"ID":"932df59a-5bc9-4c43-a4ad-55dc8b516b7b","Type":"ContainerStarted","Data":"b305940539dfa8d3c0e5caa0765ac5c29d11eb7ee1d788aeeea9388dcae454a4"} Nov 01 06:02:46.370316 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:46.370300 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:02:46.370494 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:46.370461 5040 patch_prober.go:29] interesting pod/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 01 06:02:46.370544 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:46.370528 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" podUID="932df59a-5bc9-4c43-a4ad-55dc8b516b7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 01 06:02:46.479557 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:46.479502 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddf4e87-zcpb9" podStartSLOduration=2.479474735 podStartE2EDuration="2.479474735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:02:46.476858423 +0000 UTC m=+1045.131975605" watchObservedRunningTime="2026-11-01 06:02:46.479474735 +0000 UTC m=+1045.134591901" Nov 01 06:02:47.369078 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:47.369042 5040 generic.go:356] "Generic (PLEG): container finished" podID="0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5" containerID="02038dfa70f16ef0855d455cfd5d3a85a2dfa62284139c2cc7846cb58d2cf9dc" exitCode=0 Nov 01 06:02:47.369459 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:47.369121 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddf4e87-zcpb9" event={"ID":"0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5","Type":"ContainerDied","Data":"02038dfa70f16ef0855d455cfd5d3a85a2dfa62284139c2cc7846cb58d2cf9dc"} Nov 01 06:02:47.370762 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:47.370742 5040 patch_prober.go:29] interesting pod/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 01 06:02:47.370810 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:47.370794 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" podUID="932df59a-5bc9-4c43-a4ad-55dc8b516b7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 01 06:02:48.370971 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:48.370937 5040 patch_prober.go:29] interesting pod/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 01 06:02:48.371427 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:48.370997 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" podUID="932df59a-5bc9-4c43-a4ad-55dc8b516b7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 01 06:02:48.552203 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:48.552179 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddf4e87-zcpb9" Nov 01 06:02:48.725960 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:48.725857 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5-kube-api-access-f82z8\" (UniqueName: \"kubernetes.io/projected/0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5-kube-api-access-f82z8\") pod \"0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5\" (UID: \"0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5\") " Nov 01 06:02:48.725960 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:48.725905 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5-maas-api-openshift-ca\") pod \"0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5\" (UID: \"0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5\") " Nov 01 06:02:48.726216 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:48.725973 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5-maas-api-xks-ca\") pod \"0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5\" (UID: \"0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5\") " Nov 01 06:02:48.726347 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:48.726304 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5-maas-api-openshift-ca" pod "0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5" (UID: "0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:02:48.726571 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:48.726541 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:02:48.728319 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:48.728291 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5-maas-api-xks-ca" pod "0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5" (UID: "0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:02:48.728964 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:48.728934 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5-kube-api-access-f82z8" pod "0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5" (UID: "0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5"). InnerVolumeSpecName "kube-api-access-f82z8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:02:48.827695 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:48.827657 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f82z8\" (UniqueName: \"kubernetes.io/projected/0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5-kube-api-access-f82z8\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:02:48.827695 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:48.827691 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:02:49.370845 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:49.370811 5040 patch_prober.go:29] interesting pod/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 01 06:02:49.371015 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:49.370894 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" podUID="932df59a-5bc9-4c43-a4ad-55dc8b516b7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 01 06:02:49.388139 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:49.388101 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddf4e87-zcpb9" event={"ID":"0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5","Type":"ContainerDied","Data":"b8cbbca9bbf7d2798974e9d5fb256d5f54a52f2e8b13563b05859560ab354926"} Nov 01 06:02:49.388139 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:49.388128 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddf4e87-zcpb9" Nov 01 06:02:49.388344 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:49.388182 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8cbbca9bbf7d2798974e9d5fb256d5f54a52f2e8b13563b05859560ab354926" Nov 01 06:02:50.370989 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:50.370956 5040 patch_prober.go:29] interesting pod/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 01 06:02:50.371169 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:50.371009 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" podUID="932df59a-5bc9-4c43-a4ad-55dc8b516b7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 01 06:02:51.370300 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:51.370268 5040 patch_prober.go:29] interesting pod/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 01 06:02:51.370537 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:51.370319 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" podUID="932df59a-5bc9-4c43-a4ad-55dc8b516b7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 01 06:02:51.987612 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:51.987580 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:02:52.374243 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:52.374208 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:02:52.408131 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:52.408106 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:02:53.488043 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:53.488006 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs"] Nov 01 06:02:53.489266 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:53.489238 5040 cpu_manager.go:414] "containerMap: removing container" podUID="0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5" containerName="revoke-keys" Nov 01 06:02:53.489397 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:53.489384 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5" containerName="revoke-keys" Nov 01 06:02:53.489633 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:53.489618 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5" containerName="revoke-keys" Nov 01 06:02:53.499771 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:53.499744 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" Nov 01 06:02:53.565742 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:53.565709 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-f73e89-openshift-default-dockercfg-tqlnq\"" Nov 01 06:02:53.582025 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:53.581981 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xtfq\" (UniqueName: \"kubernetes.io/projected/8d7c546f-f34b-4824-b012-962a5e7f68c6-kube-api-access-5xtfq\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs\" (UID: \"8d7c546f-f34b-4824-b012-962a5e7f68c6\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" Nov 01 06:02:53.582291 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:53.582267 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7c546f-f34b-4824-b012-962a5e7f68c6-workload-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs\" (UID: \"8d7c546f-f34b-4824-b012-962a5e7f68c6\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" Nov 01 06:02:53.582414 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:53.582399 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d7c546f-f34b-4824-b012-962a5e7f68c6-istio-envoy\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs\" (UID: \"8d7c546f-f34b-4824-b012-962a5e7f68c6\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" Nov 01 06:02:53.582552 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:53.582534 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7c546f-f34b-4824-b012-962a5e7f68c6-credential-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs\" (UID: \"8d7c546f-f34b-4824-b012-962a5e7f68c6\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" Nov 01 06:02:53.582716 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:53.582702 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d7c546f-f34b-4824-b012-962a5e7f68c6-workload-certs\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs\" (UID: \"8d7c546f-f34b-4824-b012-962a5e7f68c6\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" Nov 01 06:02:53.582850 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:53.582832 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d7c546f-f34b-4824-b012-962a5e7f68c6-istio-podinfo\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs\" (UID: \"8d7c546f-f34b-4824-b012-962a5e7f68c6\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" Nov 01 06:02:53.582979 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:53.582964 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d7c546f-f34b-4824-b012-962a5e7f68c6-istiod-ca-cert\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs\" (UID: \"8d7c546f-f34b-4824-b012-962a5e7f68c6\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" Nov 01 06:02:53.583097 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:53.583084 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d7c546f-f34b-4824-b012-962a5e7f68c6-istio-token\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs\" (UID: \"8d7c546f-f34b-4824-b012-962a5e7f68c6\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" Nov 01 06:02:53.583210 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:53.583194 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d7c546f-f34b-4824-b012-962a5e7f68c6-istio-data\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs\" (UID: \"8d7c546f-f34b-4824-b012-962a5e7f68c6\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" Nov 01 06:02:53.643505 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:53.629664 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs"] Nov 01 06:02:53.769499 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:53.765980 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d7c546f-f34b-4824-b012-962a5e7f68c6-istiod-ca-cert\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs\" (UID: \"8d7c546f-f34b-4824-b012-962a5e7f68c6\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" Nov 01 06:02:53.769499 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:53.766045 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d7c546f-f34b-4824-b012-962a5e7f68c6-istio-token\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs\" (UID: \"8d7c546f-f34b-4824-b012-962a5e7f68c6\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" Nov 01 06:02:53.769499 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:53.766075 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d7c546f-f34b-4824-b012-962a5e7f68c6-istio-data\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs\" (UID: \"8d7c546f-f34b-4824-b012-962a5e7f68c6\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" Nov 01 06:02:53.769499 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:53.766140 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xtfq\" (UniqueName: \"kubernetes.io/projected/8d7c546f-f34b-4824-b012-962a5e7f68c6-kube-api-access-5xtfq\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs\" (UID: \"8d7c546f-f34b-4824-b012-962a5e7f68c6\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" Nov 01 06:02:53.769499 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:53.766169 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7c546f-f34b-4824-b012-962a5e7f68c6-workload-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs\" (UID: \"8d7c546f-f34b-4824-b012-962a5e7f68c6\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" Nov 01 06:02:53.769499 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:53.766195 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d7c546f-f34b-4824-b012-962a5e7f68c6-istio-envoy\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs\" (UID: \"8d7c546f-f34b-4824-b012-962a5e7f68c6\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" Nov 01 06:02:53.769499 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:53.766227 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7c546f-f34b-4824-b012-962a5e7f68c6-credential-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs\" (UID: \"8d7c546f-f34b-4824-b012-962a5e7f68c6\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" Nov 01 06:02:53.769499 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:53.766304 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d7c546f-f34b-4824-b012-962a5e7f68c6-workload-certs\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs\" (UID: \"8d7c546f-f34b-4824-b012-962a5e7f68c6\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" Nov 01 06:02:53.769499 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:53.766348 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d7c546f-f34b-4824-b012-962a5e7f68c6-istio-podinfo\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs\" (UID: \"8d7c546f-f34b-4824-b012-962a5e7f68c6\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" Nov 01 06:02:53.798534 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:53.781789 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d7c546f-f34b-4824-b012-962a5e7f68c6-istiod-ca-cert\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs\" (UID: \"8d7c546f-f34b-4824-b012-962a5e7f68c6\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" Nov 01 06:02:53.799419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:53.790010 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d7c546f-f34b-4824-b012-962a5e7f68c6-istio-podinfo\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs\" (UID: \"8d7c546f-f34b-4824-b012-962a5e7f68c6\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" Nov 01 06:02:53.799627 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:53.790266 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7c546f-f34b-4824-b012-962a5e7f68c6-credential-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs\" (UID: \"8d7c546f-f34b-4824-b012-962a5e7f68c6\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" Nov 01 06:02:53.799751 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:53.790496 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d7c546f-f34b-4824-b012-962a5e7f68c6-workload-certs\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs\" (UID: \"8d7c546f-f34b-4824-b012-962a5e7f68c6\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" Nov 01 06:02:53.799860 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:53.790702 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d7c546f-f34b-4824-b012-962a5e7f68c6-istio-data\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs\" (UID: \"8d7c546f-f34b-4824-b012-962a5e7f68c6\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" Nov 01 06:02:53.799964 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:53.790919 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7c546f-f34b-4824-b012-962a5e7f68c6-workload-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs\" (UID: \"8d7c546f-f34b-4824-b012-962a5e7f68c6\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" Nov 01 06:02:53.832500 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:53.828761 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d7c546f-f34b-4824-b012-962a5e7f68c6-istio-envoy\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs\" (UID: \"8d7c546f-f34b-4824-b012-962a5e7f68c6\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" Nov 01 06:02:53.863509 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:53.861576 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d7c546f-f34b-4824-b012-962a5e7f68c6-istio-token\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs\" (UID: \"8d7c546f-f34b-4824-b012-962a5e7f68c6\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" Nov 01 06:02:53.867819 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:53.867795 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xtfq\" (UniqueName: \"kubernetes.io/projected/8d7c546f-f34b-4824-b012-962a5e7f68c6-kube-api-access-5xtfq\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs\" (UID: \"8d7c546f-f34b-4824-b012-962a5e7f68c6\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" Nov 01 06:02:54.149530 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:54.149399 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" Nov 01 06:02:54.605248 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:54.605224 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs"] Nov 01 06:02:54.606096 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:02:54.606074 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d7c546f_f34b_4824_b012_962a5e7f68c6.slice/crio-c7834f9edf9745e5639ee3f3aa8bcdf6965521c9a8603e2d960a74a38a22fbb5 WatchSource:0}: Error finding container c7834f9edf9745e5639ee3f3aa8bcdf6965521c9a8603e2d960a74a38a22fbb5: Status 404 returned error can't find the container with id c7834f9edf9745e5639ee3f3aa8bcdf6965521c9a8603e2d960a74a38a22fbb5 Nov 01 06:02:54.608212 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:54.608180 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:02:54.608303 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:54.608242 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:02:54.608303 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:54.608282 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:02:55.430213 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:55.430172 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" event={"ID":"8d7c546f-f34b-4824-b012-962a5e7f68c6","Type":"ContainerStarted","Data":"25047227c2789d294c9a09cb93001f4ad4745caf8d974ed0730b686e2f4d5a41"} Nov 01 06:02:55.430426 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:55.430230 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" event={"ID":"8d7c546f-f34b-4824-b012-962a5e7f68c6","Type":"ContainerStarted","Data":"c7834f9edf9745e5639ee3f3aa8bcdf6965521c9a8603e2d960a74a38a22fbb5"} Nov 01 06:02:55.658620 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:55.658569 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" podStartSLOduration=2.658555108 podStartE2EDuration="2.658555108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:02:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:02:55.605980379 +0000 UTC m=+1054.261097546" watchObservedRunningTime="2026-11-01 06:02:55.658555108 +0000 UTC m=+1054.313672276" Nov 01 06:02:55.718693 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:55.718597 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2e981c2-68tqn"] Nov 01 06:02:55.723198 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:55.723174 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e981c2-68tqn" Nov 01 06:02:55.772195 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:55.772164 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2e981c2-68tqn"] Nov 01 06:02:55.776321 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:55.776278 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:02:55.776445 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:55.776413 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:02:55.776536 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:55.776429 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:02:55.776592 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:55.776532 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:02:55.784992 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:55.784967 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71284f4a-dfa0-410c-9cf2-f5753d6f21e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2e981c2-68tqn\" (UID: \"71284f4a-dfa0-410c-9cf2-f5753d6f21e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e981c2-68tqn" Nov 01 06:02:55.785101 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:55.785008 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8zc6\" (UniqueName: \"kubernetes.io/projected/71284f4a-dfa0-410c-9cf2-f5753d6f21e8-kube-api-access-x8zc6\") pod \"maas-api-revoke-sub-e2e981c2-68tqn\" (UID: \"71284f4a-dfa0-410c-9cf2-f5753d6f21e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e981c2-68tqn" Nov 01 06:02:55.785217 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:55.785196 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71284f4a-dfa0-410c-9cf2-f5753d6f21e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2e981c2-68tqn\" (UID: \"71284f4a-dfa0-410c-9cf2-f5753d6f21e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e981c2-68tqn" Nov 01 06:02:55.886229 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:55.886186 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71284f4a-dfa0-410c-9cf2-f5753d6f21e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2e981c2-68tqn\" (UID: \"71284f4a-dfa0-410c-9cf2-f5753d6f21e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e981c2-68tqn" Nov 01 06:02:55.886434 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:55.886287 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71284f4a-dfa0-410c-9cf2-f5753d6f21e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2e981c2-68tqn\" (UID: \"71284f4a-dfa0-410c-9cf2-f5753d6f21e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e981c2-68tqn" Nov 01 06:02:55.886434 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:55.886310 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8zc6\" (UniqueName: \"kubernetes.io/projected/71284f4a-dfa0-410c-9cf2-f5753d6f21e8-kube-api-access-x8zc6\") pod \"maas-api-revoke-sub-e2e981c2-68tqn\" (UID: \"71284f4a-dfa0-410c-9cf2-f5753d6f21e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e981c2-68tqn" Nov 01 06:02:55.887022 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:55.887000 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71284f4a-dfa0-410c-9cf2-f5753d6f21e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2e981c2-68tqn\" (UID: \"71284f4a-dfa0-410c-9cf2-f5753d6f21e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e981c2-68tqn" Nov 01 06:02:55.888641 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:55.888623 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71284f4a-dfa0-410c-9cf2-f5753d6f21e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2e981c2-68tqn\" (UID: \"71284f4a-dfa0-410c-9cf2-f5753d6f21e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e981c2-68tqn" Nov 01 06:02:55.947828 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:55.947796 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8zc6\" (UniqueName: \"kubernetes.io/projected/71284f4a-dfa0-410c-9cf2-f5753d6f21e8-kube-api-access-x8zc6\") pod \"maas-api-revoke-sub-e2e981c2-68tqn\" (UID: \"71284f4a-dfa0-410c-9cf2-f5753d6f21e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e981c2-68tqn" Nov 01 06:02:56.036980 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:56.036944 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e981c2-68tqn" Nov 01 06:02:56.152046 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:56.152015 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" Nov 01 06:02:56.153448 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:56.153420 5040 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 01 06:02:56.153665 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:56.153468 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" podUID="8d7c546f-f34b-4824-b012-962a5e7f68c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 01 06:02:56.422227 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:56.422193 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2e981c2-68tqn"] Nov 01 06:02:56.423374 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:02:56.423346 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71284f4a_dfa0_410c_9cf2_f5753d6f21e8.slice/crio-31345ed6406ba08ca2223b76037e529735eaa26d9ddc25532a4428d3aeeec370 WatchSource:0}: Error finding container 31345ed6406ba08ca2223b76037e529735eaa26d9ddc25532a4428d3aeeec370: Status 404 returned error can't find the container with id 31345ed6406ba08ca2223b76037e529735eaa26d9ddc25532a4428d3aeeec370 Nov 01 06:02:56.437509 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:56.437465 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e981c2-68tqn" event={"ID":"71284f4a-dfa0-410c-9cf2-f5753d6f21e8","Type":"ContainerStarted","Data":"31345ed6406ba08ca2223b76037e529735eaa26d9ddc25532a4428d3aeeec370"} Nov 01 06:02:57.151237 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:57.151150 5040 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 01 06:02:57.151237 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:57.151207 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" podUID="8d7c546f-f34b-4824-b012-962a5e7f68c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 01 06:02:57.452592 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:57.452466 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e981c2-68tqn" event={"ID":"71284f4a-dfa0-410c-9cf2-f5753d6f21e8","Type":"ContainerStarted","Data":"c573b02578890d4b1897e484b6dc64c118365d7af77faebd65a60d76933ba09e"} Nov 01 06:02:57.578915 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:57.578858 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e981c2-68tqn" podStartSLOduration=2.578845747 podStartE2EDuration="2.578845747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:02:57.578507815 +0000 UTC m=+1056.233624979" watchObservedRunningTime="2026-11-01 06:02:57.578845747 +0000 UTC m=+1056.233962914" Nov 01 06:02:58.151462 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:58.151425 5040 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 01 06:02:58.151884 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:58.151506 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" podUID="8d7c546f-f34b-4824-b012-962a5e7f68c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 01 06:02:58.460160 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:58.460067 5040 generic.go:356] "Generic (PLEG): container finished" podID="71284f4a-dfa0-410c-9cf2-f5753d6f21e8" containerID="c573b02578890d4b1897e484b6dc64c118365d7af77faebd65a60d76933ba09e" exitCode=0 Nov 01 06:02:58.460334 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:58.460152 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e981c2-68tqn" event={"ID":"71284f4a-dfa0-410c-9cf2-f5753d6f21e8","Type":"ContainerDied","Data":"c573b02578890d4b1897e484b6dc64c118365d7af77faebd65a60d76933ba09e"} Nov 01 06:02:59.151184 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:59.151152 5040 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 01 06:02:59.151346 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:59.151203 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" podUID="8d7c546f-f34b-4824-b012-962a5e7f68c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 01 06:02:59.644700 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:59.644679 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e981c2-68tqn" Nov 01 06:02:59.748254 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:59.748228 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71284f4a-dfa0-410c-9cf2-f5753d6f21e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71284f4a-dfa0-410c-9cf2-f5753d6f21e8-maas-api-openshift-ca\") pod \"71284f4a-dfa0-410c-9cf2-f5753d6f21e8\" (UID: \"71284f4a-dfa0-410c-9cf2-f5753d6f21e8\") " Nov 01 06:02:59.748423 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:59.748364 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71284f4a-dfa0-410c-9cf2-f5753d6f21e8-kube-api-access-x8zc6\" (UniqueName: \"kubernetes.io/projected/71284f4a-dfa0-410c-9cf2-f5753d6f21e8-kube-api-access-x8zc6\") pod \"71284f4a-dfa0-410c-9cf2-f5753d6f21e8\" (UID: \"71284f4a-dfa0-410c-9cf2-f5753d6f21e8\") " Nov 01 06:02:59.748474 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:59.748427 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71284f4a-dfa0-410c-9cf2-f5753d6f21e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71284f4a-dfa0-410c-9cf2-f5753d6f21e8-maas-api-xks-ca\") pod \"71284f4a-dfa0-410c-9cf2-f5753d6f21e8\" (UID: \"71284f4a-dfa0-410c-9cf2-f5753d6f21e8\") " Nov 01 06:02:59.748640 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:59.748617 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71284f4a-dfa0-410c-9cf2-f5753d6f21e8-maas-api-openshift-ca" pod "71284f4a-dfa0-410c-9cf2-f5753d6f21e8" (UID: "71284f4a-dfa0-410c-9cf2-f5753d6f21e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:02:59.748808 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:59.748785 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71284f4a-dfa0-410c-9cf2-f5753d6f21e8-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:02:59.750467 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:59.750436 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71284f4a-dfa0-410c-9cf2-f5753d6f21e8-maas-api-xks-ca" pod "71284f4a-dfa0-410c-9cf2-f5753d6f21e8" (UID: "71284f4a-dfa0-410c-9cf2-f5753d6f21e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:02:59.750577 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:59.750508 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71284f4a-dfa0-410c-9cf2-f5753d6f21e8-kube-api-access-x8zc6" pod "71284f4a-dfa0-410c-9cf2-f5753d6f21e8" (UID: "71284f4a-dfa0-410c-9cf2-f5753d6f21e8"). InnerVolumeSpecName "kube-api-access-x8zc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:02:59.849785 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:59.849750 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8zc6\" (UniqueName: \"kubernetes.io/projected/71284f4a-dfa0-410c-9cf2-f5753d6f21e8-kube-api-access-x8zc6\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:02:59.849785 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:02:59.849781 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71284f4a-dfa0-410c-9cf2-f5753d6f21e8-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:00.151637 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:00.151549 5040 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 01 06:03:00.151637 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:00.151605 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" podUID="8d7c546f-f34b-4824-b012-962a5e7f68c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 01 06:03:00.473299 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:00.473207 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e981c2-68tqn" Nov 01 06:03:00.473299 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:00.473233 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e981c2-68tqn" event={"ID":"71284f4a-dfa0-410c-9cf2-f5753d6f21e8","Type":"ContainerDied","Data":"31345ed6406ba08ca2223b76037e529735eaa26d9ddc25532a4428d3aeeec370"} Nov 01 06:03:00.473299 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:00.473289 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31345ed6406ba08ca2223b76037e529735eaa26d9ddc25532a4428d3aeeec370" Nov 01 06:03:01.140951 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:01.140911 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daeedd40-96gjv"] Nov 01 06:03:01.141669 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:01.141648 5040 cpu_manager.go:414] "containerMap: removing container" podUID="71284f4a-dfa0-410c-9cf2-f5753d6f21e8" containerName="revoke-keys" Nov 01 06:03:01.141733 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:01.141673 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71284f4a-dfa0-410c-9cf2-f5753d6f21e8" containerName="revoke-keys" Nov 01 06:03:01.141820 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:01.141806 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="71284f4a-dfa0-410c-9cf2-f5753d6f21e8" containerName="revoke-keys" Nov 01 06:03:01.145821 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:01.145781 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daeedd40-96gjv" Nov 01 06:03:01.151617 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:01.151584 5040 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 01 06:03:01.151757 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:01.151688 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" podUID="8d7c546f-f34b-4824-b012-962a5e7f68c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 01 06:03:01.200391 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:01.200359 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daeedd40-96gjv"] Nov 01 06:03:01.249970 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:01.247469 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:03:01.249970 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:01.247639 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:03:01.249970 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:01.247720 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:03:01.249970 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:01.247766 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:03:01.263861 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:01.263835 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ecf787f-f866-4daa-9354-db17829557e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daeedd40-96gjv\" (UID: \"5ecf787f-f866-4daa-9354-db17829557e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daeedd40-96gjv" Nov 01 06:03:01.263999 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:01.263876 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5hcx\" (UniqueName: \"kubernetes.io/projected/5ecf787f-f866-4daa-9354-db17829557e1-kube-api-access-c5hcx\") pod \"maas-api-revoke-sub-daeedd40-96gjv\" (UID: \"5ecf787f-f866-4daa-9354-db17829557e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daeedd40-96gjv" Nov 01 06:03:01.264129 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:01.264104 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ecf787f-f866-4daa-9354-db17829557e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daeedd40-96gjv\" (UID: \"5ecf787f-f866-4daa-9354-db17829557e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daeedd40-96gjv" Nov 01 06:03:01.365193 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:01.365158 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ecf787f-f866-4daa-9354-db17829557e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daeedd40-96gjv\" (UID: \"5ecf787f-f866-4daa-9354-db17829557e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daeedd40-96gjv" Nov 01 06:03:01.365365 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:01.365235 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ecf787f-f866-4daa-9354-db17829557e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daeedd40-96gjv\" (UID: \"5ecf787f-f866-4daa-9354-db17829557e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daeedd40-96gjv" Nov 01 06:03:01.365365 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:01.365278 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5hcx\" (UniqueName: \"kubernetes.io/projected/5ecf787f-f866-4daa-9354-db17829557e1-kube-api-access-c5hcx\") pod \"maas-api-revoke-sub-daeedd40-96gjv\" (UID: \"5ecf787f-f866-4daa-9354-db17829557e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daeedd40-96gjv" Nov 01 06:03:01.365846 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:01.365820 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ecf787f-f866-4daa-9354-db17829557e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daeedd40-96gjv\" (UID: \"5ecf787f-f866-4daa-9354-db17829557e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daeedd40-96gjv" Nov 01 06:03:01.367568 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:01.367549 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ecf787f-f866-4daa-9354-db17829557e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daeedd40-96gjv\" (UID: \"5ecf787f-f866-4daa-9354-db17829557e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daeedd40-96gjv" Nov 01 06:03:01.427449 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:01.427363 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5hcx\" (UniqueName: \"kubernetes.io/projected/5ecf787f-f866-4daa-9354-db17829557e1-kube-api-access-c5hcx\") pod \"maas-api-revoke-sub-daeedd40-96gjv\" (UID: \"5ecf787f-f866-4daa-9354-db17829557e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daeedd40-96gjv" Nov 01 06:03:01.459239 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:01.459210 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daeedd40-96gjv" Nov 01 06:03:01.850239 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:01.850207 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daeedd40-96gjv"] Nov 01 06:03:01.852425 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:03:01.852393 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ecf787f_f866_4daa_9354_db17829557e1.slice/crio-9413267fbbd578ca65b5be6c3b34521aa23b42b43b97712f3a4754fb2bdeb4f7 WatchSource:0}: Error finding container 9413267fbbd578ca65b5be6c3b34521aa23b42b43b97712f3a4754fb2bdeb4f7: Status 404 returned error can't find the container with id 9413267fbbd578ca65b5be6c3b34521aa23b42b43b97712f3a4754fb2bdeb4f7 Nov 01 06:03:02.155617 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:02.155586 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" Nov 01 06:03:02.156079 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:02.155960 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" Nov 01 06:03:02.156563 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:02.156545 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" Nov 01 06:03:02.490795 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:02.490690 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daeedd40-96gjv" event={"ID":"5ecf787f-f866-4daa-9354-db17829557e1","Type":"ContainerStarted","Data":"ed2514b62a6d09cc013c8a0f6bdd2cf11da9ea47eb508cfae77160143163d1dc"} Nov 01 06:03:02.490795 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:02.490791 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daeedd40-96gjv" event={"ID":"5ecf787f-f866-4daa-9354-db17829557e1","Type":"ContainerStarted","Data":"9413267fbbd578ca65b5be6c3b34521aa23b42b43b97712f3a4754fb2bdeb4f7"} Nov 01 06:03:02.613495 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:02.613429 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daeedd40-96gjv" podStartSLOduration=1.6134154139999999 podStartE2EDuration="1.613415414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:03:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:03:02.610836435 +0000 UTC m=+1061.265953604" watchObservedRunningTime="2026-11-01 06:03:02.613415414 +0000 UTC m=+1061.268532580" Nov 01 06:03:03.240585 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:03.240498 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx"] Nov 01 06:03:03.244312 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:03.244294 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" Nov 01 06:03:03.351823 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:03.351791 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx"] Nov 01 06:03:03.388814 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:03.388786 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b5c7304-1fa5-495f-b821-79392bbd7376-istio-data\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx\" (UID: \"9b5c7304-1fa5-495f-b821-79392bbd7376\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" Nov 01 06:03:03.388988 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:03.388829 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5c7304-1fa5-495f-b821-79392bbd7376-credential-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx\" (UID: \"9b5c7304-1fa5-495f-b821-79392bbd7376\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" Nov 01 06:03:03.388988 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:03.388933 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b5c7304-1fa5-495f-b821-79392bbd7376-istiod-ca-cert\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx\" (UID: \"9b5c7304-1fa5-495f-b821-79392bbd7376\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" Nov 01 06:03:03.388988 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:03.388965 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b5c7304-1fa5-495f-b821-79392bbd7376-istio-podinfo\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx\" (UID: \"9b5c7304-1fa5-495f-b821-79392bbd7376\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" Nov 01 06:03:03.389130 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:03.389018 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b5c7304-1fa5-495f-b821-79392bbd7376-workload-certs\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx\" (UID: \"9b5c7304-1fa5-495f-b821-79392bbd7376\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" Nov 01 06:03:03.389130 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:03.389041 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b5c7304-1fa5-495f-b821-79392bbd7376-istio-envoy\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx\" (UID: \"9b5c7304-1fa5-495f-b821-79392bbd7376\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" Nov 01 06:03:03.389130 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:03.389068 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5c7304-1fa5-495f-b821-79392bbd7376-workload-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx\" (UID: \"9b5c7304-1fa5-495f-b821-79392bbd7376\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" Nov 01 06:03:03.389219 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:03.389186 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k9v5\" (UniqueName: \"kubernetes.io/projected/9b5c7304-1fa5-495f-b821-79392bbd7376-kube-api-access-9k9v5\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx\" (UID: \"9b5c7304-1fa5-495f-b821-79392bbd7376\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" Nov 01 06:03:03.389260 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:03.389246 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b5c7304-1fa5-495f-b821-79392bbd7376-istio-token\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx\" (UID: \"9b5c7304-1fa5-495f-b821-79392bbd7376\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" Nov 01 06:03:03.490025 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:03.489983 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b5c7304-1fa5-495f-b821-79392bbd7376-istio-data\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx\" (UID: \"9b5c7304-1fa5-495f-b821-79392bbd7376\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" Nov 01 06:03:03.490025 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:03.490030 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5c7304-1fa5-495f-b821-79392bbd7376-credential-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx\" (UID: \"9b5c7304-1fa5-495f-b821-79392bbd7376\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" Nov 01 06:03:03.490278 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:03.490061 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b5c7304-1fa5-495f-b821-79392bbd7376-istiod-ca-cert\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx\" (UID: \"9b5c7304-1fa5-495f-b821-79392bbd7376\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" Nov 01 06:03:03.490278 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:03.490086 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b5c7304-1fa5-495f-b821-79392bbd7376-istio-podinfo\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx\" (UID: \"9b5c7304-1fa5-495f-b821-79392bbd7376\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" Nov 01 06:03:03.490278 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:03.490109 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b5c7304-1fa5-495f-b821-79392bbd7376-workload-certs\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx\" (UID: \"9b5c7304-1fa5-495f-b821-79392bbd7376\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" Nov 01 06:03:03.490278 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:03.490125 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b5c7304-1fa5-495f-b821-79392bbd7376-istio-envoy\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx\" (UID: \"9b5c7304-1fa5-495f-b821-79392bbd7376\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" Nov 01 06:03:03.490278 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:03.490141 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5c7304-1fa5-495f-b821-79392bbd7376-workload-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx\" (UID: \"9b5c7304-1fa5-495f-b821-79392bbd7376\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" Nov 01 06:03:03.490278 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:03.490179 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k9v5\" (UniqueName: \"kubernetes.io/projected/9b5c7304-1fa5-495f-b821-79392bbd7376-kube-api-access-9k9v5\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx\" (UID: \"9b5c7304-1fa5-495f-b821-79392bbd7376\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" Nov 01 06:03:03.490278 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:03.490221 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b5c7304-1fa5-495f-b821-79392bbd7376-istio-token\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx\" (UID: \"9b5c7304-1fa5-495f-b821-79392bbd7376\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" Nov 01 06:03:03.490714 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:03.490507 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b5c7304-1fa5-495f-b821-79392bbd7376-istio-data\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx\" (UID: \"9b5c7304-1fa5-495f-b821-79392bbd7376\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" Nov 01 06:03:03.490714 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:03.490540 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5c7304-1fa5-495f-b821-79392bbd7376-credential-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx\" (UID: \"9b5c7304-1fa5-495f-b821-79392bbd7376\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" Nov 01 06:03:03.490714 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:03.490615 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5c7304-1fa5-495f-b821-79392bbd7376-workload-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx\" (UID: \"9b5c7304-1fa5-495f-b821-79392bbd7376\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" Nov 01 06:03:03.490870 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:03.490810 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b5c7304-1fa5-495f-b821-79392bbd7376-workload-certs\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx\" (UID: \"9b5c7304-1fa5-495f-b821-79392bbd7376\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" Nov 01 06:03:03.491128 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:03.491101 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b5c7304-1fa5-495f-b821-79392bbd7376-istiod-ca-cert\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx\" (UID: \"9b5c7304-1fa5-495f-b821-79392bbd7376\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" Nov 01 06:03:03.492804 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:03.492772 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b5c7304-1fa5-495f-b821-79392bbd7376-istio-podinfo\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx\" (UID: \"9b5c7304-1fa5-495f-b821-79392bbd7376\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" Nov 01 06:03:03.492918 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:03.492904 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b5c7304-1fa5-495f-b821-79392bbd7376-istio-envoy\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx\" (UID: \"9b5c7304-1fa5-495f-b821-79392bbd7376\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" Nov 01 06:03:03.499022 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:03.498998 5040 generic.go:356] "Generic (PLEG): container finished" podID="5ecf787f-f866-4daa-9354-db17829557e1" containerID="ed2514b62a6d09cc013c8a0f6bdd2cf11da9ea47eb508cfae77160143163d1dc" exitCode=0 Nov 01 06:03:03.499119 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:03.499084 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daeedd40-96gjv" event={"ID":"5ecf787f-f866-4daa-9354-db17829557e1","Type":"ContainerDied","Data":"ed2514b62a6d09cc013c8a0f6bdd2cf11da9ea47eb508cfae77160143163d1dc"} Nov 01 06:03:03.552049 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:03.552015 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b5c7304-1fa5-495f-b821-79392bbd7376-istio-token\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx\" (UID: \"9b5c7304-1fa5-495f-b821-79392bbd7376\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" Nov 01 06:03:03.552225 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:03.552147 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k9v5\" (UniqueName: \"kubernetes.io/projected/9b5c7304-1fa5-495f-b821-79392bbd7376-kube-api-access-9k9v5\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx\" (UID: \"9b5c7304-1fa5-495f-b821-79392bbd7376\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" Nov 01 06:03:03.558035 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:03.557997 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" Nov 01 06:03:03.988783 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:03.988751 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx"] Nov 01 06:03:03.990107 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:03:03.990078 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b5c7304_1fa5_495f_b821_79392bbd7376.slice/crio-862e06941475ce120ee731b6a5defbefdfbcd5fb4c26f0bc48902e6155f1b4e4 WatchSource:0}: Error finding container 862e06941475ce120ee731b6a5defbefdfbcd5fb4c26f0bc48902e6155f1b4e4: Status 404 returned error can't find the container with id 862e06941475ce120ee731b6a5defbefdfbcd5fb4c26f0bc48902e6155f1b4e4 Nov 01 06:03:03.992731 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:03.992696 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:03:03.992798 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:03.992764 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:03:03.992798 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:03.992792 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:03:04.523555 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:04.523515 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" event={"ID":"9b5c7304-1fa5-495f-b821-79392bbd7376","Type":"ContainerStarted","Data":"52463959b30543947a2aa048b54f207ff3db2ac2ace1d64d954f7676c2daab19"} Nov 01 06:03:04.524057 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:04.523581 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" event={"ID":"9b5c7304-1fa5-495f-b821-79392bbd7376","Type":"ContainerStarted","Data":"862e06941475ce120ee731b6a5defbefdfbcd5fb4c26f0bc48902e6155f1b4e4"} Nov 01 06:03:04.558827 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:04.558787 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" Nov 01 06:03:04.649363 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:04.649301 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" podStartSLOduration=1.6492820240000001 podStartE2EDuration="1.649282024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:03:04.644066589 +0000 UTC m=+1063.299183756" watchObservedRunningTime="2026-11-01 06:03:04.649282024 +0000 UTC m=+1063.304399192" Nov 01 06:03:04.692959 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:04.692925 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daeedd40-96gjv" Nov 01 06:03:04.824294 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:04.821215 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ecf787f-f866-4daa-9354-db17829557e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ecf787f-f866-4daa-9354-db17829557e1-maas-api-openshift-ca\") pod \"5ecf787f-f866-4daa-9354-db17829557e1\" (UID: \"5ecf787f-f866-4daa-9354-db17829557e1\") " Nov 01 06:03:04.824294 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:04.821328 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ecf787f-f866-4daa-9354-db17829557e1-kube-api-access-c5hcx\" (UniqueName: \"kubernetes.io/projected/5ecf787f-f866-4daa-9354-db17829557e1-kube-api-access-c5hcx\") pod \"5ecf787f-f866-4daa-9354-db17829557e1\" (UID: \"5ecf787f-f866-4daa-9354-db17829557e1\") " Nov 01 06:03:04.824294 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:04.821404 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ecf787f-f866-4daa-9354-db17829557e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ecf787f-f866-4daa-9354-db17829557e1-maas-api-xks-ca\") pod \"5ecf787f-f866-4daa-9354-db17829557e1\" (UID: \"5ecf787f-f866-4daa-9354-db17829557e1\") " Nov 01 06:03:04.824294 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:04.822416 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ecf787f-f866-4daa-9354-db17829557e1-maas-api-openshift-ca" pod "5ecf787f-f866-4daa-9354-db17829557e1" (UID: "5ecf787f-f866-4daa-9354-db17829557e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:03:04.825352 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:04.825310 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ecf787f-f866-4daa-9354-db17829557e1-maas-api-xks-ca" pod "5ecf787f-f866-4daa-9354-db17829557e1" (UID: "5ecf787f-f866-4daa-9354-db17829557e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:03:04.826594 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:04.826565 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ecf787f-f866-4daa-9354-db17829557e1-kube-api-access-c5hcx" pod "5ecf787f-f866-4daa-9354-db17829557e1" (UID: "5ecf787f-f866-4daa-9354-db17829557e1"). InnerVolumeSpecName "kube-api-access-c5hcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:03:04.922377 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:04.922343 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ecf787f-f866-4daa-9354-db17829557e1-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:04.922377 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:04.922374 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5hcx\" (UniqueName: \"kubernetes.io/projected/5ecf787f-f866-4daa-9354-db17829557e1-kube-api-access-c5hcx\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:04.922377 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:04.922384 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ecf787f-f866-4daa-9354-db17829557e1-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:05.531919 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:05.531879 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daeedd40-96gjv" Nov 01 06:03:05.532581 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:05.532180 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daeedd40-96gjv" event={"ID":"5ecf787f-f866-4daa-9354-db17829557e1","Type":"ContainerDied","Data":"9413267fbbd578ca65b5be6c3b34521aa23b42b43b97712f3a4754fb2bdeb4f7"} Nov 01 06:03:05.532581 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:05.532278 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9413267fbbd578ca65b5be6c3b34521aa23b42b43b97712f3a4754fb2bdeb4f7" Nov 01 06:03:05.559953 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:05.559924 5040 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 01 06:03:05.560102 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:05.559982 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" podUID="9b5c7304-1fa5-495f-b821-79392bbd7376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 01 06:03:06.558460 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:06.558426 5040 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 01 06:03:06.558902 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:06.558497 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" podUID="9b5c7304-1fa5-495f-b821-79392bbd7376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 01 06:03:06.780002 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:06.779963 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf"] Nov 01 06:03:06.780712 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:06.780691 5040 cpu_manager.go:414] "containerMap: removing container" podUID="5ecf787f-f866-4daa-9354-db17829557e1" containerName="revoke-keys" Nov 01 06:03:06.780776 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:06.780715 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ecf787f-f866-4daa-9354-db17829557e1" containerName="revoke-keys" Nov 01 06:03:06.780850 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:06.780838 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ecf787f-f866-4daa-9354-db17829557e1" containerName="revoke-keys" Nov 01 06:03:06.784964 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:06.784938 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:03:06.840721 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:06.840626 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-58d523-openshift-default-dockercfg-5n4sp\"" Nov 01 06:03:06.863512 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:06.849683 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94ltf\" (UniqueName: \"kubernetes.io/projected/a325a624-c22a-4425-9002-ad68a3c22314-kube-api-access-94ltf\") pod \"e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf\" (UID: \"a325a624-c22a-4425-9002-ad68a3c22314\") " pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:03:06.863874 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:06.863846 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a325a624-c22a-4425-9002-ad68a3c22314-istio-data\") pod \"e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf\" (UID: \"a325a624-c22a-4425-9002-ad68a3c22314\") " pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:03:06.864030 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:06.864017 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a325a624-c22a-4425-9002-ad68a3c22314-istio-podinfo\") pod \"e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf\" (UID: \"a325a624-c22a-4425-9002-ad68a3c22314\") " pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:03:06.864140 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:06.864127 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a325a624-c22a-4425-9002-ad68a3c22314-workload-socket\") pod \"e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf\" (UID: \"a325a624-c22a-4425-9002-ad68a3c22314\") " pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:03:06.864244 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:06.864232 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a325a624-c22a-4425-9002-ad68a3c22314-istio-token\") pod \"e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf\" (UID: \"a325a624-c22a-4425-9002-ad68a3c22314\") " pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:03:06.864375 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:06.864362 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a325a624-c22a-4425-9002-ad68a3c22314-istio-envoy\") pod \"e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf\" (UID: \"a325a624-c22a-4425-9002-ad68a3c22314\") " pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:03:06.864513 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:06.864495 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a325a624-c22a-4425-9002-ad68a3c22314-workload-certs\") pod \"e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf\" (UID: \"a325a624-c22a-4425-9002-ad68a3c22314\") " pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:03:06.864668 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:06.864655 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a325a624-c22a-4425-9002-ad68a3c22314-istiod-ca-cert\") pod \"e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf\" (UID: \"a325a624-c22a-4425-9002-ad68a3c22314\") " pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:03:06.864759 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:06.864747 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a325a624-c22a-4425-9002-ad68a3c22314-credential-socket\") pod \"e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf\" (UID: \"a325a624-c22a-4425-9002-ad68a3c22314\") " pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:03:06.959766 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:06.959728 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf"] Nov 01 06:03:06.967052 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:06.967019 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a325a624-c22a-4425-9002-ad68a3c22314-istio-envoy\") pod \"e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf\" (UID: \"a325a624-c22a-4425-9002-ad68a3c22314\") " pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:03:06.967236 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:06.967075 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a325a624-c22a-4425-9002-ad68a3c22314-workload-certs\") pod \"e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf\" (UID: \"a325a624-c22a-4425-9002-ad68a3c22314\") " pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:03:06.967236 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:06.967133 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a325a624-c22a-4425-9002-ad68a3c22314-istiod-ca-cert\") pod \"e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf\" (UID: \"a325a624-c22a-4425-9002-ad68a3c22314\") " pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:03:06.967236 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:06.967158 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a325a624-c22a-4425-9002-ad68a3c22314-credential-socket\") pod \"e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf\" (UID: \"a325a624-c22a-4425-9002-ad68a3c22314\") " pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:03:06.967413 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:06.967243 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94ltf\" (UniqueName: \"kubernetes.io/projected/a325a624-c22a-4425-9002-ad68a3c22314-kube-api-access-94ltf\") pod \"e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf\" (UID: \"a325a624-c22a-4425-9002-ad68a3c22314\") " pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:03:06.967413 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:06.967273 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a325a624-c22a-4425-9002-ad68a3c22314-istio-data\") pod \"e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf\" (UID: \"a325a624-c22a-4425-9002-ad68a3c22314\") " pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:03:06.967413 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:06.967322 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a325a624-c22a-4425-9002-ad68a3c22314-istio-podinfo\") pod \"e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf\" (UID: \"a325a624-c22a-4425-9002-ad68a3c22314\") " pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:03:06.967413 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:06.967355 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a325a624-c22a-4425-9002-ad68a3c22314-workload-socket\") pod \"e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf\" (UID: \"a325a624-c22a-4425-9002-ad68a3c22314\") " pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:03:06.967413 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:06.967390 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a325a624-c22a-4425-9002-ad68a3c22314-istio-token\") pod \"e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf\" (UID: \"a325a624-c22a-4425-9002-ad68a3c22314\") " pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:03:06.969936 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:06.969909 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a325a624-c22a-4425-9002-ad68a3c22314-credential-socket\") pod \"e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf\" (UID: \"a325a624-c22a-4425-9002-ad68a3c22314\") " pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:03:06.970340 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:06.970315 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a325a624-c22a-4425-9002-ad68a3c22314-workload-certs\") pod \"e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf\" (UID: \"a325a624-c22a-4425-9002-ad68a3c22314\") " pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:03:06.970640 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:06.970613 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a325a624-c22a-4425-9002-ad68a3c22314-workload-socket\") pod \"e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf\" (UID: \"a325a624-c22a-4425-9002-ad68a3c22314\") " pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:03:06.971022 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:06.970999 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a325a624-c22a-4425-9002-ad68a3c22314-istio-data\") pod \"e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf\" (UID: \"a325a624-c22a-4425-9002-ad68a3c22314\") " pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:03:06.971384 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:06.971358 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a325a624-c22a-4425-9002-ad68a3c22314-istiod-ca-cert\") pod \"e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf\" (UID: \"a325a624-c22a-4425-9002-ad68a3c22314\") " pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:03:06.973219 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:06.973198 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a325a624-c22a-4425-9002-ad68a3c22314-istio-podinfo\") pod \"e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf\" (UID: \"a325a624-c22a-4425-9002-ad68a3c22314\") " pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:03:06.982011 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:06.981986 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a325a624-c22a-4425-9002-ad68a3c22314-istio-envoy\") pod \"e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf\" (UID: \"a325a624-c22a-4425-9002-ad68a3c22314\") " pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:03:07.038718 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:07.038684 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94ltf\" (UniqueName: \"kubernetes.io/projected/a325a624-c22a-4425-9002-ad68a3c22314-kube-api-access-94ltf\") pod \"e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf\" (UID: \"a325a624-c22a-4425-9002-ad68a3c22314\") " pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:03:07.039015 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:07.038993 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a325a624-c22a-4425-9002-ad68a3c22314-istio-token\") pod \"e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf\" (UID: \"a325a624-c22a-4425-9002-ad68a3c22314\") " pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:03:07.115998 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:07.115904 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:03:07.558705 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:07.558670 5040 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 01 06:03:07.559096 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:07.558735 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" podUID="9b5c7304-1fa5-495f-b821-79392bbd7376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 01 06:03:07.567399 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:03:07.567365 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda325a624_c22a_4425_9002_ad68a3c22314.slice/crio-2ddd1f61ee4516f7a6a55c961388f9349e3b8c523b5cc0e5ecd726d31c77453b WatchSource:0}: Error finding container 2ddd1f61ee4516f7a6a55c961388f9349e3b8c523b5cc0e5ecd726d31c77453b: Status 404 returned error can't find the container with id 2ddd1f61ee4516f7a6a55c961388f9349e3b8c523b5cc0e5ecd726d31c77453b Nov 01 06:03:07.568258 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:07.568228 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf"] Nov 01 06:03:07.569890 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:07.569857 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:03:07.569975 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:07.569922 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:03:07.569975 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:07.569951 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:03:08.568504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:08.566358 5040 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 01 06:03:08.568504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:08.566432 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" podUID="9b5c7304-1fa5-495f-b821-79392bbd7376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 01 06:03:08.568504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:08.567105 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" event={"ID":"a325a624-c22a-4425-9002-ad68a3c22314","Type":"ContainerStarted","Data":"c390947f4f51a2b41e5f3cc13b3e0531642cc06e939c569d148ce41319b5775b"} Nov 01 06:03:08.568504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:08.567266 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" event={"ID":"a325a624-c22a-4425-9002-ad68a3c22314","Type":"ContainerStarted","Data":"2ddd1f61ee4516f7a6a55c961388f9349e3b8c523b5cc0e5ecd726d31c77453b"} Nov 01 06:03:08.698632 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:08.698571 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" podStartSLOduration=2.698536223 podStartE2EDuration="2.698536223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:03:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:03:08.694548207 +0000 UTC m=+1067.349665396" watchObservedRunningTime="2026-11-01 06:03:08.698536223 +0000 UTC m=+1067.353653390" Nov 01 06:03:09.117505 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:09.117389 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:03:09.127838 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:09.127802 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:03:09.558904 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:09.558861 5040 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 01 06:03:09.559115 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:09.558922 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" podUID="9b5c7304-1fa5-495f-b821-79392bbd7376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 01 06:03:09.574644 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:09.574593 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:03:09.576246 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:09.576223 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:03:09.979361 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:03:09.979265 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:03:10.846060 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:10.846027 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" Nov 01 06:03:10.846666 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:10.846541 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" Nov 01 06:03:10.847317 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:10.847302 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" Nov 01 06:03:12.026283 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:12.026242 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eabef2e-w47sw"] Nov 01 06:03:12.030333 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:12.030316 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eabef2e-w47sw" Nov 01 06:03:12.074800 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:12.074767 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b433b2b9-e929-436d-ae27-9d42a76cbb40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8eabef2e-w47sw\" (UID: \"b433b2b9-e929-436d-ae27-9d42a76cbb40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eabef2e-w47sw" Nov 01 06:03:12.074964 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:12.074826 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpsh6\" (UniqueName: \"kubernetes.io/projected/b433b2b9-e929-436d-ae27-9d42a76cbb40-kube-api-access-bpsh6\") pod \"maas-api-revoke-sub-8eabef2e-w47sw\" (UID: \"b433b2b9-e929-436d-ae27-9d42a76cbb40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eabef2e-w47sw" Nov 01 06:03:12.075011 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:12.074977 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b433b2b9-e929-436d-ae27-9d42a76cbb40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8eabef2e-w47sw\" (UID: \"b433b2b9-e929-436d-ae27-9d42a76cbb40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eabef2e-w47sw" Nov 01 06:03:12.088460 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:12.088425 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eabef2e-w47sw"] Nov 01 06:03:12.134548 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:12.134508 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:03:12.134548 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:12.134547 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:03:12.134771 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:12.134561 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:03:12.134771 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:12.134641 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:03:12.176333 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:12.176298 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b433b2b9-e929-436d-ae27-9d42a76cbb40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8eabef2e-w47sw\" (UID: \"b433b2b9-e929-436d-ae27-9d42a76cbb40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eabef2e-w47sw" Nov 01 06:03:12.176535 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:12.176350 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpsh6\" (UniqueName: \"kubernetes.io/projected/b433b2b9-e929-436d-ae27-9d42a76cbb40-kube-api-access-bpsh6\") pod \"maas-api-revoke-sub-8eabef2e-w47sw\" (UID: \"b433b2b9-e929-436d-ae27-9d42a76cbb40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eabef2e-w47sw" Nov 01 06:03:12.176535 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:12.176406 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b433b2b9-e929-436d-ae27-9d42a76cbb40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8eabef2e-w47sw\" (UID: \"b433b2b9-e929-436d-ae27-9d42a76cbb40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eabef2e-w47sw" Nov 01 06:03:12.177176 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:12.177152 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b433b2b9-e929-436d-ae27-9d42a76cbb40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8eabef2e-w47sw\" (UID: \"b433b2b9-e929-436d-ae27-9d42a76cbb40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eabef2e-w47sw" Nov 01 06:03:12.178736 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:12.178720 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b433b2b9-e929-436d-ae27-9d42a76cbb40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8eabef2e-w47sw\" (UID: \"b433b2b9-e929-436d-ae27-9d42a76cbb40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eabef2e-w47sw" Nov 01 06:03:12.240403 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:12.240371 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpsh6\" (UniqueName: \"kubernetes.io/projected/b433b2b9-e929-436d-ae27-9d42a76cbb40-kube-api-access-bpsh6\") pod \"maas-api-revoke-sub-8eabef2e-w47sw\" (UID: \"b433b2b9-e929-436d-ae27-9d42a76cbb40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eabef2e-w47sw" Nov 01 06:03:12.343654 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:12.343571 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eabef2e-w47sw" Nov 01 06:03:12.721371 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:12.721342 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eabef2e-w47sw"] Nov 01 06:03:12.722368 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:03:12.722337 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb433b2b9_e929_436d_ae27_9d42a76cbb40.slice/crio-2473672ebb9107553ec82ed3e20b0989fca89900960e6138a9eb5e02b510dab8 WatchSource:0}: Error finding container 2473672ebb9107553ec82ed3e20b0989fca89900960e6138a9eb5e02b510dab8: Status 404 returned error can't find the container with id 2473672ebb9107553ec82ed3e20b0989fca89900960e6138a9eb5e02b510dab8 Nov 01 06:03:13.601079 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:13.601046 5040 generic.go:356] "Generic (PLEG): container finished" podID="b433b2b9-e929-436d-ae27-9d42a76cbb40" containerID="dde2bd13c2bcf06fc881f6ae79920de0db3e0ce90f89b77b65c0dda2aa014907" exitCode=0 Nov 01 06:03:13.601565 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:13.601147 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eabef2e-w47sw" event={"ID":"b433b2b9-e929-436d-ae27-9d42a76cbb40","Type":"ContainerDied","Data":"dde2bd13c2bcf06fc881f6ae79920de0db3e0ce90f89b77b65c0dda2aa014907"} Nov 01 06:03:13.601565 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:13.601215 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eabef2e-w47sw" event={"ID":"b433b2b9-e929-436d-ae27-9d42a76cbb40","Type":"ContainerStarted","Data":"2473672ebb9107553ec82ed3e20b0989fca89900960e6138a9eb5e02b510dab8"} Nov 01 06:03:14.781921 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:14.781893 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eabef2e-w47sw" Nov 01 06:03:14.911989 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:14.911897 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b433b2b9-e929-436d-ae27-9d42a76cbb40-kube-api-access-bpsh6\" (UniqueName: \"kubernetes.io/projected/b433b2b9-e929-436d-ae27-9d42a76cbb40-kube-api-access-bpsh6\") pod \"b433b2b9-e929-436d-ae27-9d42a76cbb40\" (UID: \"b433b2b9-e929-436d-ae27-9d42a76cbb40\") " Nov 01 06:03:14.920144 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:14.920112 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b433b2b9-e929-436d-ae27-9d42a76cbb40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b433b2b9-e929-436d-ae27-9d42a76cbb40-maas-api-xks-ca\") pod \"b433b2b9-e929-436d-ae27-9d42a76cbb40\" (UID: \"b433b2b9-e929-436d-ae27-9d42a76cbb40\") " Nov 01 06:03:14.920346 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:14.920207 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b433b2b9-e929-436d-ae27-9d42a76cbb40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b433b2b9-e929-436d-ae27-9d42a76cbb40-maas-api-openshift-ca\") pod \"b433b2b9-e929-436d-ae27-9d42a76cbb40\" (UID: \"b433b2b9-e929-436d-ae27-9d42a76cbb40\") " Nov 01 06:03:14.934021 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:14.933960 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b433b2b9-e929-436d-ae27-9d42a76cbb40-maas-api-xks-ca" pod "b433b2b9-e929-436d-ae27-9d42a76cbb40" (UID: "b433b2b9-e929-436d-ae27-9d42a76cbb40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:03:14.947856 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:14.947817 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b433b2b9-e929-436d-ae27-9d42a76cbb40-maas-api-openshift-ca" pod "b433b2b9-e929-436d-ae27-9d42a76cbb40" (UID: "b433b2b9-e929-436d-ae27-9d42a76cbb40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:03:14.950040 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:14.950008 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b433b2b9-e929-436d-ae27-9d42a76cbb40-kube-api-access-bpsh6" pod "b433b2b9-e929-436d-ae27-9d42a76cbb40" (UID: "b433b2b9-e929-436d-ae27-9d42a76cbb40"). InnerVolumeSpecName "kube-api-access-bpsh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:03:15.021553 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:15.021521 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpsh6\" (UniqueName: \"kubernetes.io/projected/b433b2b9-e929-436d-ae27-9d42a76cbb40-kube-api-access-bpsh6\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:15.021553 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:15.021549 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b433b2b9-e929-436d-ae27-9d42a76cbb40-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:15.021553 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:15.021561 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b433b2b9-e929-436d-ae27-9d42a76cbb40-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:15.614047 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:15.614011 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eabef2e-w47sw" event={"ID":"b433b2b9-e929-436d-ae27-9d42a76cbb40","Type":"ContainerDied","Data":"2473672ebb9107553ec82ed3e20b0989fca89900960e6138a9eb5e02b510dab8"} Nov 01 06:03:15.614188 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:15.614065 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2473672ebb9107553ec82ed3e20b0989fca89900960e6138a9eb5e02b510dab8" Nov 01 06:03:15.614188 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:15.614025 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eabef2e-w47sw" Nov 01 06:03:24.268732 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:24.268700 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea44eee9-vcqz6"] Nov 01 06:03:24.269148 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:24.269135 5040 cpu_manager.go:414] "containerMap: removing container" podUID="b433b2b9-e929-436d-ae27-9d42a76cbb40" containerName="revoke-keys" Nov 01 06:03:24.269189 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:24.269148 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b433b2b9-e929-436d-ae27-9d42a76cbb40" containerName="revoke-keys" Nov 01 06:03:24.269256 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:24.269247 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="b433b2b9-e929-436d-ae27-9d42a76cbb40" containerName="revoke-keys" Nov 01 06:03:24.273867 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:24.273852 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea44eee9-vcqz6" Nov 01 06:03:24.330422 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:24.330386 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea44eee9-vcqz6"] Nov 01 06:03:24.335083 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:24.335050 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f434da95-fe85-4951-af26-2e9ef793a51d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea44eee9-vcqz6\" (UID: \"f434da95-fe85-4951-af26-2e9ef793a51d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea44eee9-vcqz6" Nov 01 06:03:24.335242 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:24.335178 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54gjc\" (UniqueName: \"kubernetes.io/projected/f434da95-fe85-4951-af26-2e9ef793a51d-kube-api-access-54gjc\") pod \"maas-api-revoke-sub-ea44eee9-vcqz6\" (UID: \"f434da95-fe85-4951-af26-2e9ef793a51d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea44eee9-vcqz6" Nov 01 06:03:24.335306 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:24.335270 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f434da95-fe85-4951-af26-2e9ef793a51d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea44eee9-vcqz6\" (UID: \"f434da95-fe85-4951-af26-2e9ef793a51d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea44eee9-vcqz6" Nov 01 06:03:24.376765 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:24.376736 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:03:24.376765 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:24.376739 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:03:24.376765 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:24.376745 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:03:24.377294 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:24.377276 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:03:24.436530 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:24.436455 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54gjc\" (UniqueName: \"kubernetes.io/projected/f434da95-fe85-4951-af26-2e9ef793a51d-kube-api-access-54gjc\") pod \"maas-api-revoke-sub-ea44eee9-vcqz6\" (UID: \"f434da95-fe85-4951-af26-2e9ef793a51d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea44eee9-vcqz6" Nov 01 06:03:24.436724 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:24.436576 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f434da95-fe85-4951-af26-2e9ef793a51d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea44eee9-vcqz6\" (UID: \"f434da95-fe85-4951-af26-2e9ef793a51d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea44eee9-vcqz6" Nov 01 06:03:24.436798 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:24.436729 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f434da95-fe85-4951-af26-2e9ef793a51d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea44eee9-vcqz6\" (UID: \"f434da95-fe85-4951-af26-2e9ef793a51d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea44eee9-vcqz6" Nov 01 06:03:24.437393 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:24.437369 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f434da95-fe85-4951-af26-2e9ef793a51d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea44eee9-vcqz6\" (UID: \"f434da95-fe85-4951-af26-2e9ef793a51d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea44eee9-vcqz6" Nov 01 06:03:24.439216 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:24.439189 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f434da95-fe85-4951-af26-2e9ef793a51d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea44eee9-vcqz6\" (UID: \"f434da95-fe85-4951-af26-2e9ef793a51d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea44eee9-vcqz6" Nov 01 06:03:24.498166 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:24.498136 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54gjc\" (UniqueName: \"kubernetes.io/projected/f434da95-fe85-4951-af26-2e9ef793a51d-kube-api-access-54gjc\") pod \"maas-api-revoke-sub-ea44eee9-vcqz6\" (UID: \"f434da95-fe85-4951-af26-2e9ef793a51d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea44eee9-vcqz6" Nov 01 06:03:24.587414 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:24.587329 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea44eee9-vcqz6" Nov 01 06:03:24.963113 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:24.963082 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea44eee9-vcqz6"] Nov 01 06:03:24.963783 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:03:24.963757 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf434da95_fe85_4951_af26_2e9ef793a51d.slice/crio-7ef2ca19f95623a145e9ece93b6eb42bdf7bbbe54b61afb57c2ca6ba6caf3b4a WatchSource:0}: Error finding container 7ef2ca19f95623a145e9ece93b6eb42bdf7bbbe54b61afb57c2ca6ba6caf3b4a: Status 404 returned error can't find the container with id 7ef2ca19f95623a145e9ece93b6eb42bdf7bbbe54b61afb57c2ca6ba6caf3b4a Nov 01 06:03:25.681132 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:25.681086 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea44eee9-vcqz6" event={"ID":"f434da95-fe85-4951-af26-2e9ef793a51d","Type":"ContainerStarted","Data":"ee0c00b6834f25692dda0e5ea3d0f2e4cd60170135cda8bdbb50c1abd596b96d"} Nov 01 06:03:25.681577 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:25.681154 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea44eee9-vcqz6" event={"ID":"f434da95-fe85-4951-af26-2e9ef793a51d","Type":"ContainerStarted","Data":"7ef2ca19f95623a145e9ece93b6eb42bdf7bbbe54b61afb57c2ca6ba6caf3b4a"} Nov 01 06:03:25.798075 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:25.798009 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea44eee9-vcqz6" podStartSLOduration=1.7979908230000001 podStartE2EDuration="1.797990823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:03:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:03:25.797886239 +0000 UTC m=+1084.453003407" watchObservedRunningTime="2026-11-01 06:03:25.797990823 +0000 UTC m=+1084.453107991" Nov 01 06:03:26.688676 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:26.688639 5040 generic.go:356] "Generic (PLEG): container finished" podID="f434da95-fe85-4951-af26-2e9ef793a51d" containerID="ee0c00b6834f25692dda0e5ea3d0f2e4cd60170135cda8bdbb50c1abd596b96d" exitCode=0 Nov 01 06:03:26.689064 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:26.688721 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea44eee9-vcqz6" event={"ID":"f434da95-fe85-4951-af26-2e9ef793a51d","Type":"ContainerDied","Data":"ee0c00b6834f25692dda0e5ea3d0f2e4cd60170135cda8bdbb50c1abd596b96d"} Nov 01 06:03:27.837336 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:27.837313 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea44eee9-vcqz6" Nov 01 06:03:27.973862 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:27.973774 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f434da95-fe85-4951-af26-2e9ef793a51d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f434da95-fe85-4951-af26-2e9ef793a51d-maas-api-xks-ca\") pod \"f434da95-fe85-4951-af26-2e9ef793a51d\" (UID: \"f434da95-fe85-4951-af26-2e9ef793a51d\") " Nov 01 06:03:27.973862 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:27.973848 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f434da95-fe85-4951-af26-2e9ef793a51d-kube-api-access-54gjc\" (UniqueName: \"kubernetes.io/projected/f434da95-fe85-4951-af26-2e9ef793a51d-kube-api-access-54gjc\") pod \"f434da95-fe85-4951-af26-2e9ef793a51d\" (UID: \"f434da95-fe85-4951-af26-2e9ef793a51d\") " Nov 01 06:03:27.974103 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:27.973970 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f434da95-fe85-4951-af26-2e9ef793a51d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f434da95-fe85-4951-af26-2e9ef793a51d-maas-api-openshift-ca\") pod \"f434da95-fe85-4951-af26-2e9ef793a51d\" (UID: \"f434da95-fe85-4951-af26-2e9ef793a51d\") " Nov 01 06:03:27.974359 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:27.974330 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f434da95-fe85-4951-af26-2e9ef793a51d-maas-api-openshift-ca" pod "f434da95-fe85-4951-af26-2e9ef793a51d" (UID: "f434da95-fe85-4951-af26-2e9ef793a51d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:03:27.974511 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:27.974491 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f434da95-fe85-4951-af26-2e9ef793a51d-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:27.975899 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:27.975869 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f434da95-fe85-4951-af26-2e9ef793a51d-maas-api-xks-ca" pod "f434da95-fe85-4951-af26-2e9ef793a51d" (UID: "f434da95-fe85-4951-af26-2e9ef793a51d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:03:27.975983 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:27.975969 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f434da95-fe85-4951-af26-2e9ef793a51d-kube-api-access-54gjc" pod "f434da95-fe85-4951-af26-2e9ef793a51d" (UID: "f434da95-fe85-4951-af26-2e9ef793a51d"). InnerVolumeSpecName "kube-api-access-54gjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:03:28.075320 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:28.075289 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f434da95-fe85-4951-af26-2e9ef793a51d-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:28.075320 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:28.075317 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54gjc\" (UniqueName: \"kubernetes.io/projected/f434da95-fe85-4951-af26-2e9ef793a51d-kube-api-access-54gjc\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:28.701168 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:28.701138 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea44eee9-vcqz6" Nov 01 06:03:28.701168 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:28.701142 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea44eee9-vcqz6" event={"ID":"f434da95-fe85-4951-af26-2e9ef793a51d","Type":"ContainerDied","Data":"7ef2ca19f95623a145e9ece93b6eb42bdf7bbbe54b61afb57c2ca6ba6caf3b4a"} Nov 01 06:03:28.701423 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:28.701188 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ef2ca19f95623a145e9ece93b6eb42bdf7bbbe54b61afb57c2ca6ba6caf3b4a" Nov 01 06:03:38.284094 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:38.284056 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a050afb9-dtvhj"] Nov 01 06:03:38.284616 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:38.284547 5040 cpu_manager.go:414] "containerMap: removing container" podUID="f434da95-fe85-4951-af26-2e9ef793a51d" containerName="revoke-keys" Nov 01 06:03:38.284616 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:38.284560 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f434da95-fe85-4951-af26-2e9ef793a51d" containerName="revoke-keys" Nov 01 06:03:38.284693 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:38.284635 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="f434da95-fe85-4951-af26-2e9ef793a51d" containerName="revoke-keys" Nov 01 06:03:38.287805 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:38.287791 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a050afb9-dtvhj" Nov 01 06:03:38.296948 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:38.296923 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh2jc\" (UniqueName: \"kubernetes.io/projected/17821c70-5bbf-493f-8d6c-75a217a4e4fb-kube-api-access-lh2jc\") pod \"maas-api-revoke-sub-a050afb9-dtvhj\" (UID: \"17821c70-5bbf-493f-8d6c-75a217a4e4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a050afb9-dtvhj" Nov 01 06:03:38.297075 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:38.296980 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17821c70-5bbf-493f-8d6c-75a217a4e4fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a050afb9-dtvhj\" (UID: \"17821c70-5bbf-493f-8d6c-75a217a4e4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a050afb9-dtvhj" Nov 01 06:03:38.297164 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:38.297144 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17821c70-5bbf-493f-8d6c-75a217a4e4fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a050afb9-dtvhj\" (UID: \"17821c70-5bbf-493f-8d6c-75a217a4e4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a050afb9-dtvhj" Nov 01 06:03:38.341505 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:38.341451 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:03:38.391919 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:38.391885 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:03:38.391919 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:38.391906 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:03:38.392137 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:38.392063 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:03:38.395313 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:38.395285 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a050afb9-dtvhj"] Nov 01 06:03:38.397963 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:38.397941 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh2jc\" (UniqueName: \"kubernetes.io/projected/17821c70-5bbf-493f-8d6c-75a217a4e4fb-kube-api-access-lh2jc\") pod \"maas-api-revoke-sub-a050afb9-dtvhj\" (UID: \"17821c70-5bbf-493f-8d6c-75a217a4e4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a050afb9-dtvhj" Nov 01 06:03:38.398037 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:38.398003 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17821c70-5bbf-493f-8d6c-75a217a4e4fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a050afb9-dtvhj\" (UID: \"17821c70-5bbf-493f-8d6c-75a217a4e4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a050afb9-dtvhj" Nov 01 06:03:38.398123 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:38.398109 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17821c70-5bbf-493f-8d6c-75a217a4e4fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a050afb9-dtvhj\" (UID: \"17821c70-5bbf-493f-8d6c-75a217a4e4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a050afb9-dtvhj" Nov 01 06:03:38.398770 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:38.398750 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17821c70-5bbf-493f-8d6c-75a217a4e4fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a050afb9-dtvhj\" (UID: \"17821c70-5bbf-493f-8d6c-75a217a4e4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a050afb9-dtvhj" Nov 01 06:03:38.400321 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:38.400305 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17821c70-5bbf-493f-8d6c-75a217a4e4fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a050afb9-dtvhj\" (UID: \"17821c70-5bbf-493f-8d6c-75a217a4e4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a050afb9-dtvhj" Nov 01 06:03:38.460140 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:38.460106 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh2jc\" (UniqueName: \"kubernetes.io/projected/17821c70-5bbf-493f-8d6c-75a217a4e4fb-kube-api-access-lh2jc\") pod \"maas-api-revoke-sub-a050afb9-dtvhj\" (UID: \"17821c70-5bbf-493f-8d6c-75a217a4e4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a050afb9-dtvhj" Nov 01 06:03:38.601357 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:38.601284 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a050afb9-dtvhj" Nov 01 06:03:38.863646 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:38.863574 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6"] Nov 01 06:03:38.868454 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:38.868433 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" Nov 01 06:03:38.903097 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:38.903067 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba373b4-ae6e-45af-9833-766ae34813d5-workload-certs\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6\" (UID: \"eba373b4-ae6e-45af-9833-766ae34813d5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" Nov 01 06:03:38.903247 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:38.903119 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba373b4-ae6e-45af-9833-766ae34813d5-istiod-ca-cert\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6\" (UID: \"eba373b4-ae6e-45af-9833-766ae34813d5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" Nov 01 06:03:38.903247 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:38.903228 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dztvz\" (UniqueName: \"kubernetes.io/projected/eba373b4-ae6e-45af-9833-766ae34813d5-kube-api-access-dztvz\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6\" (UID: \"eba373b4-ae6e-45af-9833-766ae34813d5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" Nov 01 06:03:38.903328 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:38.903263 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba373b4-ae6e-45af-9833-766ae34813d5-istio-envoy\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6\" (UID: \"eba373b4-ae6e-45af-9833-766ae34813d5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" Nov 01 06:03:38.903363 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:38.903342 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba373b4-ae6e-45af-9833-766ae34813d5-istio-podinfo\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6\" (UID: \"eba373b4-ae6e-45af-9833-766ae34813d5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" Nov 01 06:03:38.903417 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:38.903403 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eba373b4-ae6e-45af-9833-766ae34813d5-istio-token\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6\" (UID: \"eba373b4-ae6e-45af-9833-766ae34813d5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" Nov 01 06:03:38.903507 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:38.903475 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba373b4-ae6e-45af-9833-766ae34813d5-workload-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6\" (UID: \"eba373b4-ae6e-45af-9833-766ae34813d5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" Nov 01 06:03:38.903559 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:38.903525 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba373b4-ae6e-45af-9833-766ae34813d5-istio-data\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6\" (UID: \"eba373b4-ae6e-45af-9833-766ae34813d5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" Nov 01 06:03:38.903597 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:38.903574 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba373b4-ae6e-45af-9833-766ae34813d5-credential-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6\" (UID: \"eba373b4-ae6e-45af-9833-766ae34813d5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" Nov 01 06:03:38.959417 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:38.959381 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6"] Nov 01 06:03:38.977409 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:38.977375 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a050afb9-dtvhj"] Nov 01 06:03:38.977985 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:03:38.977959 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17821c70_5bbf_493f_8d6c_75a217a4e4fb.slice/crio-2b9e520c541a2ba8c16b1f4813fa650e9e12b2fcb180f4a3f6703f6e4a04a62b WatchSource:0}: Error finding container 2b9e520c541a2ba8c16b1f4813fa650e9e12b2fcb180f4a3f6703f6e4a04a62b: Status 404 returned error can't find the container with id 2b9e520c541a2ba8c16b1f4813fa650e9e12b2fcb180f4a3f6703f6e4a04a62b Nov 01 06:03:39.004231 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:39.004206 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dztvz\" (UniqueName: \"kubernetes.io/projected/eba373b4-ae6e-45af-9833-766ae34813d5-kube-api-access-dztvz\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6\" (UID: \"eba373b4-ae6e-45af-9833-766ae34813d5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" Nov 01 06:03:39.004364 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:39.004244 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba373b4-ae6e-45af-9833-766ae34813d5-istio-envoy\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6\" (UID: \"eba373b4-ae6e-45af-9833-766ae34813d5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" Nov 01 06:03:39.004364 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:39.004282 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba373b4-ae6e-45af-9833-766ae34813d5-istio-podinfo\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6\" (UID: \"eba373b4-ae6e-45af-9833-766ae34813d5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" Nov 01 06:03:39.004364 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:39.004343 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eba373b4-ae6e-45af-9833-766ae34813d5-istio-token\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6\" (UID: \"eba373b4-ae6e-45af-9833-766ae34813d5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" Nov 01 06:03:39.004364 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:39.004360 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba373b4-ae6e-45af-9833-766ae34813d5-workload-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6\" (UID: \"eba373b4-ae6e-45af-9833-766ae34813d5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" Nov 01 06:03:39.004673 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:39.004381 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba373b4-ae6e-45af-9833-766ae34813d5-istio-data\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6\" (UID: \"eba373b4-ae6e-45af-9833-766ae34813d5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" Nov 01 06:03:39.004673 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:39.004412 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba373b4-ae6e-45af-9833-766ae34813d5-credential-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6\" (UID: \"eba373b4-ae6e-45af-9833-766ae34813d5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" Nov 01 06:03:39.004673 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:39.004442 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba373b4-ae6e-45af-9833-766ae34813d5-workload-certs\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6\" (UID: \"eba373b4-ae6e-45af-9833-766ae34813d5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" Nov 01 06:03:39.004673 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:39.004473 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba373b4-ae6e-45af-9833-766ae34813d5-istiod-ca-cert\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6\" (UID: \"eba373b4-ae6e-45af-9833-766ae34813d5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" Nov 01 06:03:39.004938 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:39.004830 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba373b4-ae6e-45af-9833-766ae34813d5-workload-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6\" (UID: \"eba373b4-ae6e-45af-9833-766ae34813d5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" Nov 01 06:03:39.004938 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:39.004895 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba373b4-ae6e-45af-9833-766ae34813d5-istio-data\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6\" (UID: \"eba373b4-ae6e-45af-9833-766ae34813d5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" Nov 01 06:03:39.005043 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:39.004956 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba373b4-ae6e-45af-9833-766ae34813d5-credential-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6\" (UID: \"eba373b4-ae6e-45af-9833-766ae34813d5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" Nov 01 06:03:39.005207 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:39.005186 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba373b4-ae6e-45af-9833-766ae34813d5-istiod-ca-cert\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6\" (UID: \"eba373b4-ae6e-45af-9833-766ae34813d5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" Nov 01 06:03:39.005207 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:39.005196 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba373b4-ae6e-45af-9833-766ae34813d5-workload-certs\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6\" (UID: \"eba373b4-ae6e-45af-9833-766ae34813d5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" Nov 01 06:03:39.006564 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:39.006545 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba373b4-ae6e-45af-9833-766ae34813d5-istio-envoy\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6\" (UID: \"eba373b4-ae6e-45af-9833-766ae34813d5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" Nov 01 06:03:39.006833 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:39.006813 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba373b4-ae6e-45af-9833-766ae34813d5-istio-podinfo\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6\" (UID: \"eba373b4-ae6e-45af-9833-766ae34813d5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" Nov 01 06:03:39.066354 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:39.066329 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eba373b4-ae6e-45af-9833-766ae34813d5-istio-token\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6\" (UID: \"eba373b4-ae6e-45af-9833-766ae34813d5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" Nov 01 06:03:39.066496 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:39.066389 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dztvz\" (UniqueName: \"kubernetes.io/projected/eba373b4-ae6e-45af-9833-766ae34813d5-kube-api-access-dztvz\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6\" (UID: \"eba373b4-ae6e-45af-9833-766ae34813d5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" Nov 01 06:03:39.182210 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:39.182126 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" Nov 01 06:03:39.612232 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:39.612200 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6"] Nov 01 06:03:39.612661 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:03:39.612636 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeba373b4_ae6e_45af_9833_766ae34813d5.slice/crio-0e632d79b2b7a4ab6d92730c39a8760f977c1f593455d4d5669dbea2b192b01a WatchSource:0}: Error finding container 0e632d79b2b7a4ab6d92730c39a8760f977c1f593455d4d5669dbea2b192b01a: Status 404 returned error can't find the container with id 0e632d79b2b7a4ab6d92730c39a8760f977c1f593455d4d5669dbea2b192b01a Nov 01 06:03:39.615028 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:39.614996 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:03:39.615124 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:39.615062 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:03:39.615124 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:39.615102 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:03:39.772796 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:39.772755 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a050afb9-dtvhj" event={"ID":"17821c70-5bbf-493f-8d6c-75a217a4e4fb","Type":"ContainerStarted","Data":"a36800499c5c3d8299d90485b1215eb529fc8090255ad4aff048c85b92c0cee3"} Nov 01 06:03:39.772996 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:39.772856 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a050afb9-dtvhj" event={"ID":"17821c70-5bbf-493f-8d6c-75a217a4e4fb","Type":"ContainerStarted","Data":"2b9e520c541a2ba8c16b1f4813fa650e9e12b2fcb180f4a3f6703f6e4a04a62b"} Nov 01 06:03:39.774644 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:39.774610 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" event={"ID":"eba373b4-ae6e-45af-9833-766ae34813d5","Type":"ContainerStarted","Data":"6a77c1d0d9ff606c23eee591e1ea26ecf32e7f94bb9037b135d5d8072683b86b"} Nov 01 06:03:39.774766 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:39.774663 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" event={"ID":"eba373b4-ae6e-45af-9833-766ae34813d5","Type":"ContainerStarted","Data":"0e632d79b2b7a4ab6d92730c39a8760f977c1f593455d4d5669dbea2b192b01a"} Nov 01 06:03:39.890591 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:39.890440 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a050afb9-dtvhj" podStartSLOduration=1.8904235219999999 podStartE2EDuration="1.890423522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:03:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:03:39.889896809 +0000 UTC m=+1098.545013977" watchObservedRunningTime="2026-11-01 06:03:39.890423522 +0000 UTC m=+1098.545540696" Nov 01 06:03:39.974309 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:03:39.974274 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:03:40.014106 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:40.014052 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" podStartSLOduration=2.014037758 podStartE2EDuration="2.014037758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:03:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:03:40.011020097 +0000 UTC m=+1098.666137266" watchObservedRunningTime="2026-11-01 06:03:40.014037758 +0000 UTC m=+1098.669154926" Nov 01 06:03:40.182934 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:40.182848 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" Nov 01 06:03:40.184605 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:40.184578 5040 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 01 06:03:40.184718 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:40.184624 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" podUID="eba373b4-ae6e-45af-9833-766ae34813d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 01 06:03:40.593059 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:40.593018 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbf06b98-nq2pc"] Nov 01 06:03:40.622663 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:40.622632 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf06b98-nq2pc" Nov 01 06:03:40.654187 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:40.654158 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbf06b98-nq2pc"] Nov 01 06:03:40.722378 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:40.722338 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ed195c-edee-4b54-a492-e08e45d2aa10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbf06b98-nq2pc\" (UID: \"f2ed195c-edee-4b54-a492-e08e45d2aa10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf06b98-nq2pc" Nov 01 06:03:40.722378 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:40.722382 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ed195c-edee-4b54-a492-e08e45d2aa10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbf06b98-nq2pc\" (UID: \"f2ed195c-edee-4b54-a492-e08e45d2aa10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf06b98-nq2pc" Nov 01 06:03:40.722634 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:40.722555 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns97n\" (UniqueName: \"kubernetes.io/projected/f2ed195c-edee-4b54-a492-e08e45d2aa10-kube-api-access-ns97n\") pod \"maas-api-revoke-sub-dbf06b98-nq2pc\" (UID: \"f2ed195c-edee-4b54-a492-e08e45d2aa10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf06b98-nq2pc" Nov 01 06:03:40.785978 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:40.785941 5040 generic.go:356] "Generic (PLEG): container finished" podID="17821c70-5bbf-493f-8d6c-75a217a4e4fb" containerID="a36800499c5c3d8299d90485b1215eb529fc8090255ad4aff048c85b92c0cee3" exitCode=0 Nov 01 06:03:40.786138 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:40.786023 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a050afb9-dtvhj" event={"ID":"17821c70-5bbf-493f-8d6c-75a217a4e4fb","Type":"ContainerDied","Data":"a36800499c5c3d8299d90485b1215eb529fc8090255ad4aff048c85b92c0cee3"} Nov 01 06:03:40.823381 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:40.823345 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ed195c-edee-4b54-a492-e08e45d2aa10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbf06b98-nq2pc\" (UID: \"f2ed195c-edee-4b54-a492-e08e45d2aa10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf06b98-nq2pc" Nov 01 06:03:40.823597 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:40.823549 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ed195c-edee-4b54-a492-e08e45d2aa10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbf06b98-nq2pc\" (UID: \"f2ed195c-edee-4b54-a492-e08e45d2aa10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf06b98-nq2pc" Nov 01 06:03:40.823763 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:40.823742 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns97n\" (UniqueName: \"kubernetes.io/projected/f2ed195c-edee-4b54-a492-e08e45d2aa10-kube-api-access-ns97n\") pod \"maas-api-revoke-sub-dbf06b98-nq2pc\" (UID: \"f2ed195c-edee-4b54-a492-e08e45d2aa10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf06b98-nq2pc" Nov 01 06:03:40.824230 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:40.824203 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ed195c-edee-4b54-a492-e08e45d2aa10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbf06b98-nq2pc\" (UID: \"f2ed195c-edee-4b54-a492-e08e45d2aa10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf06b98-nq2pc" Nov 01 06:03:40.825809 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:40.825791 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ed195c-edee-4b54-a492-e08e45d2aa10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbf06b98-nq2pc\" (UID: \"f2ed195c-edee-4b54-a492-e08e45d2aa10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf06b98-nq2pc" Nov 01 06:03:40.887171 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:40.887099 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns97n\" (UniqueName: \"kubernetes.io/projected/f2ed195c-edee-4b54-a492-e08e45d2aa10-kube-api-access-ns97n\") pod \"maas-api-revoke-sub-dbf06b98-nq2pc\" (UID: \"f2ed195c-edee-4b54-a492-e08e45d2aa10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf06b98-nq2pc" Nov 01 06:03:40.935487 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:40.935438 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf06b98-nq2pc" Nov 01 06:03:41.183541 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:41.183428 5040 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 01 06:03:41.183541 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:41.183494 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" podUID="eba373b4-ae6e-45af-9833-766ae34813d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 01 06:03:41.323253 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:03:41.323219 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2ed195c_edee_4b54_a492_e08e45d2aa10.slice/crio-f70fb70e3324b9f34e17218e2dac8b4fa5407e535d4f4841fbd26e252d510b86 WatchSource:0}: Error finding container f70fb70e3324b9f34e17218e2dac8b4fa5407e535d4f4841fbd26e252d510b86: Status 404 returned error can't find the container with id f70fb70e3324b9f34e17218e2dac8b4fa5407e535d4f4841fbd26e252d510b86 Nov 01 06:03:41.323651 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:41.323631 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbf06b98-nq2pc"] Nov 01 06:03:41.820554 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:41.820128 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf06b98-nq2pc" event={"ID":"f2ed195c-edee-4b54-a492-e08e45d2aa10","Type":"ContainerStarted","Data":"3b064ba62af995d265c7d18099d34e1d7ffa18a1f26f1ef833222080326da7a4"} Nov 01 06:03:41.820554 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:41.820184 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf06b98-nq2pc" event={"ID":"f2ed195c-edee-4b54-a492-e08e45d2aa10","Type":"ContainerStarted","Data":"f70fb70e3324b9f34e17218e2dac8b4fa5407e535d4f4841fbd26e252d510b86"} Nov 01 06:03:41.952036 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:41.951979 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf06b98-nq2pc" podStartSLOduration=1.9519657179999998 podStartE2EDuration="1.951965718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:03:41.950254761 +0000 UTC m=+1100.605371929" watchObservedRunningTime="2026-11-01 06:03:41.951965718 +0000 UTC m=+1100.607082884" Nov 01 06:03:41.967654 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:41.967631 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a050afb9-dtvhj" Nov 01 06:03:42.043775 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:42.043742 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17821c70-5bbf-493f-8d6c-75a217a4e4fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17821c70-5bbf-493f-8d6c-75a217a4e4fb-maas-api-xks-ca\") pod \"17821c70-5bbf-493f-8d6c-75a217a4e4fb\" (UID: \"17821c70-5bbf-493f-8d6c-75a217a4e4fb\") " Nov 01 06:03:42.043979 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:42.043804 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17821c70-5bbf-493f-8d6c-75a217a4e4fb-kube-api-access-lh2jc\" (UniqueName: \"kubernetes.io/projected/17821c70-5bbf-493f-8d6c-75a217a4e4fb-kube-api-access-lh2jc\") pod \"17821c70-5bbf-493f-8d6c-75a217a4e4fb\" (UID: \"17821c70-5bbf-493f-8d6c-75a217a4e4fb\") " Nov 01 06:03:42.045898 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:42.045871 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17821c70-5bbf-493f-8d6c-75a217a4e4fb-kube-api-access-lh2jc" pod "17821c70-5bbf-493f-8d6c-75a217a4e4fb" (UID: "17821c70-5bbf-493f-8d6c-75a217a4e4fb"). InnerVolumeSpecName "kube-api-access-lh2jc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:03:42.046184 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:42.046162 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17821c70-5bbf-493f-8d6c-75a217a4e4fb-maas-api-xks-ca" pod "17821c70-5bbf-493f-8d6c-75a217a4e4fb" (UID: "17821c70-5bbf-493f-8d6c-75a217a4e4fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:03:42.144537 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:42.144433 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17821c70-5bbf-493f-8d6c-75a217a4e4fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17821c70-5bbf-493f-8d6c-75a217a4e4fb-maas-api-openshift-ca\") pod \"17821c70-5bbf-493f-8d6c-75a217a4e4fb\" (UID: \"17821c70-5bbf-493f-8d6c-75a217a4e4fb\") " Nov 01 06:03:42.144701 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:42.144679 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17821c70-5bbf-493f-8d6c-75a217a4e4fb-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:42.144701 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:42.144695 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh2jc\" (UniqueName: \"kubernetes.io/projected/17821c70-5bbf-493f-8d6c-75a217a4e4fb-kube-api-access-lh2jc\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:42.144883 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:42.144800 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17821c70-5bbf-493f-8d6c-75a217a4e4fb-maas-api-openshift-ca" pod "17821c70-5bbf-493f-8d6c-75a217a4e4fb" (UID: "17821c70-5bbf-493f-8d6c-75a217a4e4fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:03:42.182792 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:42.182758 5040 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 01 06:03:42.182950 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:42.182810 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" podUID="eba373b4-ae6e-45af-9833-766ae34813d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 01 06:03:42.245646 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:42.245605 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17821c70-5bbf-493f-8d6c-75a217a4e4fb-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:42.827705 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:42.827667 5040 generic.go:356] "Generic (PLEG): container finished" podID="f2ed195c-edee-4b54-a492-e08e45d2aa10" containerID="3b064ba62af995d265c7d18099d34e1d7ffa18a1f26f1ef833222080326da7a4" exitCode=0 Nov 01 06:03:42.828090 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:42.827762 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf06b98-nq2pc" event={"ID":"f2ed195c-edee-4b54-a492-e08e45d2aa10","Type":"ContainerDied","Data":"3b064ba62af995d265c7d18099d34e1d7ffa18a1f26f1ef833222080326da7a4"} Nov 01 06:03:42.829121 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:42.829102 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a050afb9-dtvhj" event={"ID":"17821c70-5bbf-493f-8d6c-75a217a4e4fb","Type":"ContainerDied","Data":"2b9e520c541a2ba8c16b1f4813fa650e9e12b2fcb180f4a3f6703f6e4a04a62b"} Nov 01 06:03:42.829163 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:42.829141 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b9e520c541a2ba8c16b1f4813fa650e9e12b2fcb180f4a3f6703f6e4a04a62b" Nov 01 06:03:42.829218 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:42.829187 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a050afb9-dtvhj" Nov 01 06:03:43.182705 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:43.182617 5040 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 01 06:03:43.182705 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:43.182673 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" podUID="eba373b4-ae6e-45af-9833-766ae34813d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 01 06:03:44.032068 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:44.032040 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf06b98-nq2pc" Nov 01 06:03:44.059566 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:44.059426 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2ed195c-edee-4b54-a492-e08e45d2aa10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ed195c-edee-4b54-a492-e08e45d2aa10-maas-api-openshift-ca\") pod \"f2ed195c-edee-4b54-a492-e08e45d2aa10\" (UID: \"f2ed195c-edee-4b54-a492-e08e45d2aa10\") " Nov 01 06:03:44.059566 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:44.059519 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2ed195c-edee-4b54-a492-e08e45d2aa10-kube-api-access-ns97n\" (UniqueName: \"kubernetes.io/projected/f2ed195c-edee-4b54-a492-e08e45d2aa10-kube-api-access-ns97n\") pod \"f2ed195c-edee-4b54-a492-e08e45d2aa10\" (UID: \"f2ed195c-edee-4b54-a492-e08e45d2aa10\") " Nov 01 06:03:44.059975 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:44.059749 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ed195c-edee-4b54-a492-e08e45d2aa10-maas-api-openshift-ca" pod "f2ed195c-edee-4b54-a492-e08e45d2aa10" (UID: "f2ed195c-edee-4b54-a492-e08e45d2aa10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:03:44.060187 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:44.060108 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2ed195c-edee-4b54-a492-e08e45d2aa10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ed195c-edee-4b54-a492-e08e45d2aa10-maas-api-xks-ca\") pod \"f2ed195c-edee-4b54-a492-e08e45d2aa10\" (UID: \"f2ed195c-edee-4b54-a492-e08e45d2aa10\") " Nov 01 06:03:44.060557 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:44.060536 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ed195c-edee-4b54-a492-e08e45d2aa10-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:44.061671 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:44.061629 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ed195c-edee-4b54-a492-e08e45d2aa10-kube-api-access-ns97n" pod "f2ed195c-edee-4b54-a492-e08e45d2aa10" (UID: "f2ed195c-edee-4b54-a492-e08e45d2aa10"). InnerVolumeSpecName "kube-api-access-ns97n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:03:44.061671 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:44.061639 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2ed195c-edee-4b54-a492-e08e45d2aa10-maas-api-xks-ca" pod "f2ed195c-edee-4b54-a492-e08e45d2aa10" (UID: "f2ed195c-edee-4b54-a492-e08e45d2aa10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:03:44.161261 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:44.161182 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns97n\" (UniqueName: \"kubernetes.io/projected/f2ed195c-edee-4b54-a492-e08e45d2aa10-kube-api-access-ns97n\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:44.161261 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:44.161212 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ed195c-edee-4b54-a492-e08e45d2aa10-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:44.182649 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:44.182616 5040 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 01 06:03:44.182809 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:44.182671 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" podUID="eba373b4-ae6e-45af-9833-766ae34813d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 01 06:03:44.843363 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:44.843327 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf06b98-nq2pc" event={"ID":"f2ed195c-edee-4b54-a492-e08e45d2aa10","Type":"ContainerDied","Data":"f70fb70e3324b9f34e17218e2dac8b4fa5407e535d4f4841fbd26e252d510b86"} Nov 01 06:03:44.843363 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:44.843362 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf06b98-nq2pc" Nov 01 06:03:44.843600 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:44.843378 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f70fb70e3324b9f34e17218e2dac8b4fa5407e535d4f4841fbd26e252d510b86" Nov 01 06:03:45.183383 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:45.183017 5040 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 01 06:03:45.183383 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:45.183084 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" podUID="eba373b4-ae6e-45af-9833-766ae34813d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 01 06:03:46.182709 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:46.182673 5040 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 01 06:03:46.182872 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:46.182743 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" podUID="eba373b4-ae6e-45af-9833-766ae34813d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 01 06:03:48.110778 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:48.110741 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" Nov 01 06:03:48.111267 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:48.111033 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" Nov 01 06:03:48.112021 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:48.111999 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" Nov 01 06:03:53.985695 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:53.985653 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb"] Nov 01 06:03:53.986354 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:53.986332 5040 cpu_manager.go:414] "containerMap: removing container" podUID="f2ed195c-edee-4b54-a492-e08e45d2aa10" containerName="revoke-keys" Nov 01 06:03:53.986446 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:53.986356 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2ed195c-edee-4b54-a492-e08e45d2aa10" containerName="revoke-keys" Nov 01 06:03:53.986446 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:53.986394 5040 cpu_manager.go:414] "containerMap: removing container" podUID="17821c70-5bbf-493f-8d6c-75a217a4e4fb" containerName="revoke-keys" Nov 01 06:03:53.986446 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:53.986403 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17821c70-5bbf-493f-8d6c-75a217a4e4fb" containerName="revoke-keys" Nov 01 06:03:53.986643 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:53.986552 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2ed195c-edee-4b54-a492-e08e45d2aa10" containerName="revoke-keys" Nov 01 06:03:53.986643 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:53.986573 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="17821c70-5bbf-493f-8d6c-75a217a4e4fb" containerName="revoke-keys" Nov 01 06:03:53.991050 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:53.991027 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" Nov 01 06:03:54.084908 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:54.084872 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb"] Nov 01 06:03:54.085452 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:54.085420 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac11b327-8061-4422-af8d-a37b205d74cf-istio-data\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb\" (UID: \"ac11b327-8061-4422-af8d-a37b205d74cf\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" Nov 01 06:03:54.085619 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:54.085598 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac11b327-8061-4422-af8d-a37b205d74cf-istio-token\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb\" (UID: \"ac11b327-8061-4422-af8d-a37b205d74cf\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" Nov 01 06:03:54.085691 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:54.085670 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac11b327-8061-4422-af8d-a37b205d74cf-istiod-ca-cert\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb\" (UID: \"ac11b327-8061-4422-af8d-a37b205d74cf\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" Nov 01 06:03:54.085738 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:54.085723 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac11b327-8061-4422-af8d-a37b205d74cf-workload-certs\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb\" (UID: \"ac11b327-8061-4422-af8d-a37b205d74cf\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" Nov 01 06:03:54.085832 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:54.085811 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac11b327-8061-4422-af8d-a37b205d74cf-workload-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb\" (UID: \"ac11b327-8061-4422-af8d-a37b205d74cf\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" Nov 01 06:03:54.085886 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:54.085871 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac11b327-8061-4422-af8d-a37b205d74cf-istio-envoy\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb\" (UID: \"ac11b327-8061-4422-af8d-a37b205d74cf\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" Nov 01 06:03:54.085972 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:54.085955 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac11b327-8061-4422-af8d-a37b205d74cf-credential-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb\" (UID: \"ac11b327-8061-4422-af8d-a37b205d74cf\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" Nov 01 06:03:54.086051 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:54.086031 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac11b327-8061-4422-af8d-a37b205d74cf-istio-podinfo\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb\" (UID: \"ac11b327-8061-4422-af8d-a37b205d74cf\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" Nov 01 06:03:54.086157 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:54.086135 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fbkq\" (UniqueName: \"kubernetes.io/projected/ac11b327-8061-4422-af8d-a37b205d74cf-kube-api-access-6fbkq\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb\" (UID: \"ac11b327-8061-4422-af8d-a37b205d74cf\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" Nov 01 06:03:54.186766 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:54.186724 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac11b327-8061-4422-af8d-a37b205d74cf-credential-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb\" (UID: \"ac11b327-8061-4422-af8d-a37b205d74cf\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" Nov 01 06:03:54.186945 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:54.186779 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac11b327-8061-4422-af8d-a37b205d74cf-istio-podinfo\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb\" (UID: \"ac11b327-8061-4422-af8d-a37b205d74cf\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" Nov 01 06:03:54.186945 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:54.186847 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fbkq\" (UniqueName: \"kubernetes.io/projected/ac11b327-8061-4422-af8d-a37b205d74cf-kube-api-access-6fbkq\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb\" (UID: \"ac11b327-8061-4422-af8d-a37b205d74cf\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" Nov 01 06:03:54.186945 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:54.186895 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac11b327-8061-4422-af8d-a37b205d74cf-istio-data\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb\" (UID: \"ac11b327-8061-4422-af8d-a37b205d74cf\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" Nov 01 06:03:54.187090 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:54.186947 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac11b327-8061-4422-af8d-a37b205d74cf-istio-token\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb\" (UID: \"ac11b327-8061-4422-af8d-a37b205d74cf\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" Nov 01 06:03:54.187090 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:54.186973 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac11b327-8061-4422-af8d-a37b205d74cf-istiod-ca-cert\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb\" (UID: \"ac11b327-8061-4422-af8d-a37b205d74cf\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" Nov 01 06:03:54.187090 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:54.187012 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac11b327-8061-4422-af8d-a37b205d74cf-workload-certs\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb\" (UID: \"ac11b327-8061-4422-af8d-a37b205d74cf\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" Nov 01 06:03:54.187242 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:54.187217 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac11b327-8061-4422-af8d-a37b205d74cf-credential-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb\" (UID: \"ac11b327-8061-4422-af8d-a37b205d74cf\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" Nov 01 06:03:54.187302 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:54.187263 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac11b327-8061-4422-af8d-a37b205d74cf-istio-data\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb\" (UID: \"ac11b327-8061-4422-af8d-a37b205d74cf\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" Nov 01 06:03:54.187302 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:54.187124 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac11b327-8061-4422-af8d-a37b205d74cf-workload-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb\" (UID: \"ac11b327-8061-4422-af8d-a37b205d74cf\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" Nov 01 06:03:54.187412 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:54.187326 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac11b327-8061-4422-af8d-a37b205d74cf-istio-envoy\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb\" (UID: \"ac11b327-8061-4422-af8d-a37b205d74cf\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" Nov 01 06:03:54.187412 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:54.187369 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac11b327-8061-4422-af8d-a37b205d74cf-workload-certs\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb\" (UID: \"ac11b327-8061-4422-af8d-a37b205d74cf\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" Nov 01 06:03:54.187693 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:54.187676 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac11b327-8061-4422-af8d-a37b205d74cf-workload-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb\" (UID: \"ac11b327-8061-4422-af8d-a37b205d74cf\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" Nov 01 06:03:54.187901 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:54.187872 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac11b327-8061-4422-af8d-a37b205d74cf-istiod-ca-cert\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb\" (UID: \"ac11b327-8061-4422-af8d-a37b205d74cf\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" Nov 01 06:03:54.189391 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:54.189361 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac11b327-8061-4422-af8d-a37b205d74cf-istio-podinfo\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb\" (UID: \"ac11b327-8061-4422-af8d-a37b205d74cf\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" Nov 01 06:03:54.189532 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:54.189451 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac11b327-8061-4422-af8d-a37b205d74cf-istio-envoy\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb\" (UID: \"ac11b327-8061-4422-af8d-a37b205d74cf\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" Nov 01 06:03:54.248745 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:54.248716 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac11b327-8061-4422-af8d-a37b205d74cf-istio-token\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb\" (UID: \"ac11b327-8061-4422-af8d-a37b205d74cf\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" Nov 01 06:03:54.248939 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:54.248917 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fbkq\" (UniqueName: \"kubernetes.io/projected/ac11b327-8061-4422-af8d-a37b205d74cf-kube-api-access-6fbkq\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb\" (UID: \"ac11b327-8061-4422-af8d-a37b205d74cf\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" Nov 01 06:03:54.304439 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:54.304409 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" Nov 01 06:03:54.734303 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:54.734272 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb"] Nov 01 06:03:54.734734 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:03:54.734711 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac11b327_8061_4422_af8d_a37b205d74cf.slice/crio-6b39865837b8f3870753f5e77e57afb52987b91eb2ebdae28bbe56e8f795ef6f WatchSource:0}: Error finding container 6b39865837b8f3870753f5e77e57afb52987b91eb2ebdae28bbe56e8f795ef6f: Status 404 returned error can't find the container with id 6b39865837b8f3870753f5e77e57afb52987b91eb2ebdae28bbe56e8f795ef6f Nov 01 06:03:54.737111 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:54.737081 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:03:54.737169 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:54.737144 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:03:54.737205 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:54.737172 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:03:54.909382 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:54.909347 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" event={"ID":"ac11b327-8061-4422-af8d-a37b205d74cf","Type":"ContainerStarted","Data":"eb313244f18a2917805528aa97996609ad8fe94ceaaba137a498f8eb79798246"} Nov 01 06:03:54.909581 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:54.909400 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" event={"ID":"ac11b327-8061-4422-af8d-a37b205d74cf","Type":"ContainerStarted","Data":"6b39865837b8f3870753f5e77e57afb52987b91eb2ebdae28bbe56e8f795ef6f"} Nov 01 06:03:55.034883 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:55.034820 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" podStartSLOduration=2.034799333 podStartE2EDuration="2.034799333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:03:55.032851065 +0000 UTC m=+1113.687968232" watchObservedRunningTime="2026-11-01 06:03:55.034799333 +0000 UTC m=+1113.689916503" Nov 01 06:03:55.310576 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:55.307874 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" Nov 01 06:03:55.315984 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:55.315945 5040 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 01 06:03:55.316255 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:55.316225 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" podUID="ac11b327-8061-4422-af8d-a37b205d74cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 01 06:03:56.319033 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:56.307739 5040 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 01 06:03:56.319033 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:56.318582 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" podUID="ac11b327-8061-4422-af8d-a37b205d74cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 01 06:03:57.318507 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:57.311785 5040 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 01 06:03:57.318507 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:57.311846 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" podUID="ac11b327-8061-4422-af8d-a37b205d74cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 01 06:03:58.315942 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:58.315905 5040 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 01 06:03:58.316603 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:58.316570 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" podUID="ac11b327-8061-4422-af8d-a37b205d74cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 01 06:03:59.318527 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:59.311861 5040 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 01 06:03:59.318527 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:03:59.311934 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" podUID="ac11b327-8061-4422-af8d-a37b205d74cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 01 06:04:00.308504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:00.306888 5040 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 01 06:04:00.308504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:00.306951 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" podUID="ac11b327-8061-4422-af8d-a37b205d74cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 01 06:04:01.305107 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:01.305068 5040 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 01 06:04:01.305519 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:01.305129 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" podUID="ac11b327-8061-4422-af8d-a37b205d74cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 01 06:04:02.310019 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:02.309975 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" Nov 01 06:04:02.310402 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:02.310358 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" Nov 01 06:04:02.311205 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:02.311183 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" Nov 01 06:04:07.425293 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:07.425259 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fc2a0b8-649s8"] Nov 01 06:04:07.430212 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:07.430190 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc2a0b8-649s8" Nov 01 06:04:07.486251 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:07.486218 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fc2a0b8-649s8"] Nov 01 06:04:07.530318 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:07.530283 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fc2a0b8-649s8\" (UID: \"9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc2a0b8-649s8" Nov 01 06:04:07.530591 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:07.530561 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq9jv\" (UniqueName: \"kubernetes.io/projected/9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8-kube-api-access-wq9jv\") pod \"maas-api-revoke-sub-6fc2a0b8-649s8\" (UID: \"9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc2a0b8-649s8" Nov 01 06:04:07.530657 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:07.530630 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fc2a0b8-649s8\" (UID: \"9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc2a0b8-649s8" Nov 01 06:04:07.532356 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:07.532328 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:04:07.532356 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:07.532355 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:04:07.532517 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:07.532371 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:04:07.532517 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:07.532503 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:04:07.631345 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:07.631313 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq9jv\" (UniqueName: \"kubernetes.io/projected/9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8-kube-api-access-wq9jv\") pod \"maas-api-revoke-sub-6fc2a0b8-649s8\" (UID: \"9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc2a0b8-649s8" Nov 01 06:04:07.631525 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:07.631361 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fc2a0b8-649s8\" (UID: \"9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc2a0b8-649s8" Nov 01 06:04:07.631525 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:07.631396 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fc2a0b8-649s8\" (UID: \"9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc2a0b8-649s8" Nov 01 06:04:07.632324 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:07.632297 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fc2a0b8-649s8\" (UID: \"9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc2a0b8-649s8" Nov 01 06:04:07.633791 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:07.633772 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fc2a0b8-649s8\" (UID: \"9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc2a0b8-649s8" Nov 01 06:04:07.693599 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:07.693499 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq9jv\" (UniqueName: \"kubernetes.io/projected/9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8-kube-api-access-wq9jv\") pod \"maas-api-revoke-sub-6fc2a0b8-649s8\" (UID: \"9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc2a0b8-649s8" Nov 01 06:04:07.752000 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:07.751965 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc2a0b8-649s8" Nov 01 06:04:08.148017 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:08.147981 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fc2a0b8-649s8"] Nov 01 06:04:08.148493 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:04:08.148451 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dee132c_5bc0_4b5d_95dc_eabfe5b1a2d8.slice/crio-338b1a5e40ea2f635b7b186c6d3deb37940dd25a08fc7b4092aa4435c6b2dd44 WatchSource:0}: Error finding container 338b1a5e40ea2f635b7b186c6d3deb37940dd25a08fc7b4092aa4435c6b2dd44: Status 404 returned error can't find the container with id 338b1a5e40ea2f635b7b186c6d3deb37940dd25a08fc7b4092aa4435c6b2dd44 Nov 01 06:04:09.036804 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:09.036767 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc2a0b8-649s8" event={"ID":"9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8","Type":"ContainerStarted","Data":"3b795a245719fb7a3f1b0c39db0f36700924890eba354dd5ca7ab4ad5fe264f0"} Nov 01 06:04:09.037184 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:09.036831 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc2a0b8-649s8" event={"ID":"9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8","Type":"ContainerStarted","Data":"338b1a5e40ea2f635b7b186c6d3deb37940dd25a08fc7b4092aa4435c6b2dd44"} Nov 01 06:04:09.153947 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:09.153899 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc2a0b8-649s8" podStartSLOduration=2.1538861320000002 podStartE2EDuration="2.153886132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:04:09.153278796 +0000 UTC m=+1127.808395963" watchObservedRunningTime="2026-11-01 06:04:09.153886132 +0000 UTC m=+1127.809003298" Nov 01 06:04:09.974723 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:04:09.974679 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:04:10.046015 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:10.045981 5040 generic.go:356] "Generic (PLEG): container finished" podID="9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8" containerID="3b795a245719fb7a3f1b0c39db0f36700924890eba354dd5ca7ab4ad5fe264f0" exitCode=0 Nov 01 06:04:10.046539 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:10.046111 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc2a0b8-649s8" event={"ID":"9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8","Type":"ContainerDied","Data":"3b795a245719fb7a3f1b0c39db0f36700924890eba354dd5ca7ab4ad5fe264f0"} Nov 01 06:04:10.616639 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:10.616592 5040 patch_prober.go:29] interesting pod/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 06:04:10.616817 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:10.616654 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" podUID="a325a624-c22a-4425-9002-ad68a3c22314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 06:04:11.199878 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:11.199852 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc2a0b8-649s8" Nov 01 06:04:11.277828 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:11.277792 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8-maas-api-xks-ca\") pod \"9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8\" (UID: \"9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8\") " Nov 01 06:04:11.278023 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:11.277856 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8-kube-api-access-wq9jv\" (UniqueName: \"kubernetes.io/projected/9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8-kube-api-access-wq9jv\") pod \"9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8\" (UID: \"9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8\") " Nov 01 06:04:11.278023 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:11.277920 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8-maas-api-openshift-ca\") pod \"9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8\" (UID: \"9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8\") " Nov 01 06:04:11.278290 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:11.278267 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8-maas-api-openshift-ca" pod "9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8" (UID: "9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:04:11.280041 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:11.280000 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8-maas-api-xks-ca" pod "9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8" (UID: "9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:04:11.280155 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:11.280036 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8-kube-api-access-wq9jv" pod "9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8" (UID: "9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8"). InnerVolumeSpecName "kube-api-access-wq9jv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:04:11.379998 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:11.379916 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:11.379998 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:11.379955 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:11.379998 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:11.379971 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq9jv\" (UniqueName: \"kubernetes.io/projected/9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8-kube-api-access-wq9jv\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:12.059465 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:12.059380 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc2a0b8-649s8" Nov 01 06:04:12.059465 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:12.059412 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc2a0b8-649s8" event={"ID":"9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8","Type":"ContainerDied","Data":"338b1a5e40ea2f635b7b186c6d3deb37940dd25a08fc7b4092aa4435c6b2dd44"} Nov 01 06:04:12.059709 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:12.059520 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="338b1a5e40ea2f635b7b186c6d3deb37940dd25a08fc7b4092aa4435c6b2dd44" Nov 01 06:04:12.061439 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:12.061411 5040 generic.go:356] "Generic (PLEG): container finished" podID="a325a624-c22a-4425-9002-ad68a3c22314" containerID="c390947f4f51a2b41e5f3cc13b3e0531642cc06e939c569d148ce41319b5775b" exitCode=0 Nov 01 06:04:12.061604 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:12.061447 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" event={"ID":"a325a624-c22a-4425-9002-ad68a3c22314","Type":"ContainerDied","Data":"c390947f4f51a2b41e5f3cc13b3e0531642cc06e939c569d148ce41319b5775b"} Nov 01 06:04:12.061900 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:12.061881 5040 scope.go:122] "RemoveContainer" containerID="c390947f4f51a2b41e5f3cc13b3e0531642cc06e939c569d148ce41319b5775b" Nov 01 06:04:12.063201 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:12.063173 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:04:12.063250 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:12.063242 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:04:12.063285 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:12.063273 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:04:13.068826 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:13.068790 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" event={"ID":"a325a624-c22a-4425-9002-ad68a3c22314","Type":"ContainerStarted","Data":"d9afdaaba3e9450be83ed80c1eb9d49491201ab0fabb20dabfd7468e206510b3"} Nov 01 06:04:13.118115 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:13.117690 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:04:13.118115 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:13.117820 5040 patch_prober.go:29] interesting pod/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 01 06:04:13.118115 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:13.117863 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" podUID="a325a624-c22a-4425-9002-ad68a3c22314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 01 06:04:13.917762 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:13.917725 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8c5ce51-cgh5j"] Nov 01 06:04:13.918743 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:13.918710 5040 cpu_manager.go:414] "containerMap: removing container" podUID="9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8" containerName="revoke-keys" Nov 01 06:04:13.918743 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:13.918738 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8" containerName="revoke-keys" Nov 01 06:04:13.918911 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:13.918887 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8" containerName="revoke-keys" Nov 01 06:04:13.923318 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:13.923299 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c5ce51-cgh5j" Nov 01 06:04:13.956614 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:13.956589 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8c5ce51-cgh5j"] Nov 01 06:04:14.014884 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:14.014752 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:04:14.014884 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:14.014811 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:04:14.014884 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:14.014854 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:04:14.026540 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:14.026506 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:04:14.033133 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:14.033105 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97686a71-a832-4014-83de-9c9c710f1b62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8c5ce51-cgh5j\" (UID: \"97686a71-a832-4014-83de-9c9c710f1b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c5ce51-cgh5j" Nov 01 06:04:14.033246 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:14.033172 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97686a71-a832-4014-83de-9c9c710f1b62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8c5ce51-cgh5j\" (UID: \"97686a71-a832-4014-83de-9c9c710f1b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c5ce51-cgh5j" Nov 01 06:04:14.033320 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:14.033279 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhxxt\" (UniqueName: \"kubernetes.io/projected/97686a71-a832-4014-83de-9c9c710f1b62-kube-api-access-lhxxt\") pod \"maas-api-revoke-sub-e8c5ce51-cgh5j\" (UID: \"97686a71-a832-4014-83de-9c9c710f1b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c5ce51-cgh5j" Nov 01 06:04:14.118937 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:14.118896 5040 patch_prober.go:29] interesting pod/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 01 06:04:14.119365 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:14.118959 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" podUID="a325a624-c22a-4425-9002-ad68a3c22314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 01 06:04:14.134510 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:14.134444 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97686a71-a832-4014-83de-9c9c710f1b62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8c5ce51-cgh5j\" (UID: \"97686a71-a832-4014-83de-9c9c710f1b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c5ce51-cgh5j" Nov 01 06:04:14.134665 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:14.134603 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhxxt\" (UniqueName: \"kubernetes.io/projected/97686a71-a832-4014-83de-9c9c710f1b62-kube-api-access-lhxxt\") pod \"maas-api-revoke-sub-e8c5ce51-cgh5j\" (UID: \"97686a71-a832-4014-83de-9c9c710f1b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c5ce51-cgh5j" Nov 01 06:04:14.134755 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:14.134737 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97686a71-a832-4014-83de-9c9c710f1b62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8c5ce51-cgh5j\" (UID: \"97686a71-a832-4014-83de-9c9c710f1b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c5ce51-cgh5j" Nov 01 06:04:14.135370 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:14.135351 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97686a71-a832-4014-83de-9c9c710f1b62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8c5ce51-cgh5j\" (UID: \"97686a71-a832-4014-83de-9c9c710f1b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c5ce51-cgh5j" Nov 01 06:04:14.136849 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:14.136829 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97686a71-a832-4014-83de-9c9c710f1b62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8c5ce51-cgh5j\" (UID: \"97686a71-a832-4014-83de-9c9c710f1b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c5ce51-cgh5j" Nov 01 06:04:14.219467 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:14.219374 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhxxt\" (UniqueName: \"kubernetes.io/projected/97686a71-a832-4014-83de-9c9c710f1b62-kube-api-access-lhxxt\") pod \"maas-api-revoke-sub-e8c5ce51-cgh5j\" (UID: \"97686a71-a832-4014-83de-9c9c710f1b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c5ce51-cgh5j" Nov 01 06:04:14.237584 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:14.237559 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c5ce51-cgh5j" Nov 01 06:04:14.641927 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:14.641895 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8c5ce51-cgh5j"] Nov 01 06:04:14.641927 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:04:14.641909 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97686a71_a832_4014_83de_9c9c710f1b62.slice/crio-21107de524543c88eb62fe845654c1a657e168d958549bb72b62350ea8df0d50 WatchSource:0}: Error finding container 21107de524543c88eb62fe845654c1a657e168d958549bb72b62350ea8df0d50: Status 404 returned error can't find the container with id 21107de524543c88eb62fe845654c1a657e168d958549bb72b62350ea8df0d50 Nov 01 06:04:15.082777 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:15.082736 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c5ce51-cgh5j" event={"ID":"97686a71-a832-4014-83de-9c9c710f1b62","Type":"ContainerStarted","Data":"e18ebde0483a0923a8df922da4ceca6f6d8f0d59c099a5c2c6e50f7dc69a3fb3"} Nov 01 06:04:15.082957 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:15.082797 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c5ce51-cgh5j" event={"ID":"97686a71-a832-4014-83de-9c9c710f1b62","Type":"ContainerStarted","Data":"21107de524543c88eb62fe845654c1a657e168d958549bb72b62350ea8df0d50"} Nov 01 06:04:15.118864 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:15.118829 5040 patch_prober.go:29] interesting pod/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 01 06:04:15.119038 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:15.118880 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" podUID="a325a624-c22a-4425-9002-ad68a3c22314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 01 06:04:15.219040 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:15.218953 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c5ce51-cgh5j" podStartSLOduration=2.218934798 podStartE2EDuration="2.218934798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:04:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:04:15.203729237 +0000 UTC m=+1133.858846404" watchObservedRunningTime="2026-11-01 06:04:15.218934798 +0000 UTC m=+1133.874051966" Nov 01 06:04:16.090013 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:16.089975 5040 generic.go:356] "Generic (PLEG): container finished" podID="97686a71-a832-4014-83de-9c9c710f1b62" containerID="e18ebde0483a0923a8df922da4ceca6f6d8f0d59c099a5c2c6e50f7dc69a3fb3" exitCode=0 Nov 01 06:04:16.090200 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:16.090025 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c5ce51-cgh5j" event={"ID":"97686a71-a832-4014-83de-9c9c710f1b62","Type":"ContainerDied","Data":"e18ebde0483a0923a8df922da4ceca6f6d8f0d59c099a5c2c6e50f7dc69a3fb3"} Nov 01 06:04:16.118901 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:16.118859 5040 patch_prober.go:29] interesting pod/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 01 06:04:16.119046 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:16.118938 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" podUID="a325a624-c22a-4425-9002-ad68a3c22314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 01 06:04:17.119434 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:17.118948 5040 patch_prober.go:29] interesting pod/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 01 06:04:17.119434 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:17.119025 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" podUID="a325a624-c22a-4425-9002-ad68a3c22314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 01 06:04:17.288290 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:17.288249 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c5ce51-cgh5j" Nov 01 06:04:17.471402 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:17.471296 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97686a71-a832-4014-83de-9c9c710f1b62-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97686a71-a832-4014-83de-9c9c710f1b62-maas-api-xks-ca\") pod \"97686a71-a832-4014-83de-9c9c710f1b62\" (UID: \"97686a71-a832-4014-83de-9c9c710f1b62\") " Nov 01 06:04:17.471402 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:17.471338 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97686a71-a832-4014-83de-9c9c710f1b62-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97686a71-a832-4014-83de-9c9c710f1b62-maas-api-openshift-ca\") pod \"97686a71-a832-4014-83de-9c9c710f1b62\" (UID: \"97686a71-a832-4014-83de-9c9c710f1b62\") " Nov 01 06:04:17.471402 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:17.471368 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97686a71-a832-4014-83de-9c9c710f1b62-kube-api-access-lhxxt\" (UniqueName: \"kubernetes.io/projected/97686a71-a832-4014-83de-9c9c710f1b62-kube-api-access-lhxxt\") pod \"97686a71-a832-4014-83de-9c9c710f1b62\" (UID: \"97686a71-a832-4014-83de-9c9c710f1b62\") " Nov 01 06:04:17.471754 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:17.471727 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97686a71-a832-4014-83de-9c9c710f1b62-maas-api-openshift-ca" pod "97686a71-a832-4014-83de-9c9c710f1b62" (UID: "97686a71-a832-4014-83de-9c9c710f1b62"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:04:17.473687 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:17.473650 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97686a71-a832-4014-83de-9c9c710f1b62-maas-api-xks-ca" pod "97686a71-a832-4014-83de-9c9c710f1b62" (UID: "97686a71-a832-4014-83de-9c9c710f1b62"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:04:17.473799 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:17.473733 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97686a71-a832-4014-83de-9c9c710f1b62-kube-api-access-lhxxt" pod "97686a71-a832-4014-83de-9c9c710f1b62" (UID: "97686a71-a832-4014-83de-9c9c710f1b62"). InnerVolumeSpecName "kube-api-access-lhxxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:04:17.573625 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:17.573586 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97686a71-a832-4014-83de-9c9c710f1b62-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:17.573625 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:17.573627 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97686a71-a832-4014-83de-9c9c710f1b62-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:17.573865 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:17.573648 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhxxt\" (UniqueName: \"kubernetes.io/projected/97686a71-a832-4014-83de-9c9c710f1b62-kube-api-access-lhxxt\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:18.106707 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:18.106669 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c5ce51-cgh5j" event={"ID":"97686a71-a832-4014-83de-9c9c710f1b62","Type":"ContainerDied","Data":"21107de524543c88eb62fe845654c1a657e168d958549bb72b62350ea8df0d50"} Nov 01 06:04:18.106885 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:18.106712 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c5ce51-cgh5j" Nov 01 06:04:18.106885 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:18.106743 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21107de524543c88eb62fe845654c1a657e168d958549bb72b62350ea8df0d50" Nov 01 06:04:18.117183 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:18.117158 5040 patch_prober.go:29] interesting pod/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 01 06:04:18.117299 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:18.117218 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" podUID="a325a624-c22a-4425-9002-ad68a3c22314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 01 06:04:19.121141 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:19.121109 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:04:19.121564 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:19.121397 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:04:19.122063 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:19.122047 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:04:23.007968 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:23.007928 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-695f5e6f-5886d"] Nov 01 06:04:23.008475 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:23.008418 5040 cpu_manager.go:414] "containerMap: removing container" podUID="97686a71-a832-4014-83de-9c9c710f1b62" containerName="revoke-keys" Nov 01 06:04:23.008475 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:23.008432 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97686a71-a832-4014-83de-9c9c710f1b62" containerName="revoke-keys" Nov 01 06:04:23.008590 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:23.008556 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="97686a71-a832-4014-83de-9c9c710f1b62" containerName="revoke-keys" Nov 01 06:04:23.011860 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:23.011845 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-695f5e6f-5886d" Nov 01 06:04:23.026894 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:23.026868 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7a48e16-9f97-4c4d-888e-9e5434f7233d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-695f5e6f-5886d\" (UID: \"a7a48e16-9f97-4c4d-888e-9e5434f7233d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-695f5e6f-5886d" Nov 01 06:04:23.027001 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:23.026928 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnm22\" (UniqueName: \"kubernetes.io/projected/a7a48e16-9f97-4c4d-888e-9e5434f7233d-kube-api-access-nnm22\") pod \"maas-api-revoke-sub-695f5e6f-5886d\" (UID: \"a7a48e16-9f97-4c4d-888e-9e5434f7233d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-695f5e6f-5886d" Nov 01 06:04:23.027098 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:23.027081 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7a48e16-9f97-4c4d-888e-9e5434f7233d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-695f5e6f-5886d\" (UID: \"a7a48e16-9f97-4c4d-888e-9e5434f7233d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-695f5e6f-5886d" Nov 01 06:04:23.069405 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:23.069379 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-695f5e6f-5886d"] Nov 01 06:04:23.115274 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:23.115243 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:04:23.115446 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:23.115330 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:04:23.115446 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:23.115367 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:04:23.115446 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:23.115367 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:04:23.127638 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:23.127613 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7a48e16-9f97-4c4d-888e-9e5434f7233d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-695f5e6f-5886d\" (UID: \"a7a48e16-9f97-4c4d-888e-9e5434f7233d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-695f5e6f-5886d" Nov 01 06:04:23.127753 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:23.127667 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7a48e16-9f97-4c4d-888e-9e5434f7233d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-695f5e6f-5886d\" (UID: \"a7a48e16-9f97-4c4d-888e-9e5434f7233d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-695f5e6f-5886d" Nov 01 06:04:23.127753 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:23.127723 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnm22\" (UniqueName: \"kubernetes.io/projected/a7a48e16-9f97-4c4d-888e-9e5434f7233d-kube-api-access-nnm22\") pod \"maas-api-revoke-sub-695f5e6f-5886d\" (UID: \"a7a48e16-9f97-4c4d-888e-9e5434f7233d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-695f5e6f-5886d" Nov 01 06:04:23.128408 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:23.128384 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7a48e16-9f97-4c4d-888e-9e5434f7233d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-695f5e6f-5886d\" (UID: \"a7a48e16-9f97-4c4d-888e-9e5434f7233d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-695f5e6f-5886d" Nov 01 06:04:23.130024 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:23.130004 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7a48e16-9f97-4c4d-888e-9e5434f7233d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-695f5e6f-5886d\" (UID: \"a7a48e16-9f97-4c4d-888e-9e5434f7233d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-695f5e6f-5886d" Nov 01 06:04:23.189834 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:23.189801 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnm22\" (UniqueName: \"kubernetes.io/projected/a7a48e16-9f97-4c4d-888e-9e5434f7233d-kube-api-access-nnm22\") pod \"maas-api-revoke-sub-695f5e6f-5886d\" (UID: \"a7a48e16-9f97-4c4d-888e-9e5434f7233d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-695f5e6f-5886d" Nov 01 06:04:23.325358 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:23.325273 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-695f5e6f-5886d" Nov 01 06:04:23.700033 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:04:23.700004 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7a48e16_9f97_4c4d_888e_9e5434f7233d.slice/crio-a8ce76f8376d056a807d7b0f87001810de5265ea0c2d7fc0aec80cc9b62f8790 WatchSource:0}: Error finding container a8ce76f8376d056a807d7b0f87001810de5265ea0c2d7fc0aec80cc9b62f8790: Status 404 returned error can't find the container with id a8ce76f8376d056a807d7b0f87001810de5265ea0c2d7fc0aec80cc9b62f8790 Nov 01 06:04:23.700308 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:23.700284 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-695f5e6f-5886d"] Nov 01 06:04:24.017999 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:24.017973 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg"] Nov 01 06:04:24.018375 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:24.018157 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" podUID="67d2bd89-b64a-4f9e-ba55-d9cb9caf1907" containerName="istio-proxy" containerID="cri-o://a83a4164778bda232058de40b85b77529ada83cb1cb1bba8ca179f21b9c16a97" gracePeriod=30 Nov 01 06:04:24.031020 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:24.030990 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf"] Nov 01 06:04:24.031210 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:24.031184 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" podUID="0aeeb98d-3738-4319-8531-dba6f0705696" containerName="istio-proxy" containerID="cri-o://9799a124214b067052467c5b105639c3ca2461227b57c8eeba925d26f3566501" gracePeriod=30 Nov 01 06:04:24.069229 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:24.069196 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl"] Nov 01 06:04:24.069439 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:24.069417 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" podUID="baa43c0f-ec86-4262-9bdc-773e9b11bdbb" containerName="istio-proxy" containerID="cri-o://70fd85aa5cc8403e134941f7c20cabed7f6e1c089e2995c2dcaf4da71f9eda82" gracePeriod=30 Nov 01 06:04:24.072613 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:24.072588 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h"] Nov 01 06:04:24.072766 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:24.072743 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" podUID="bba017ed-c86e-4442-a083-2a828ad624b1" containerName="istio-proxy" containerID="cri-o://2716189ca52914194cf6269d27c0fd8bb78d3d60b8c9d57a9fcc0fff13eee741" gracePeriod=30 Nov 01 06:04:24.148182 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:24.148144 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-695f5e6f-5886d" event={"ID":"a7a48e16-9f97-4c4d-888e-9e5434f7233d","Type":"ContainerStarted","Data":"5ca903bd862220e6f744b5fb89c73db0d6f5addc4d7c765b8dfb8112ab2ef4d6"} Nov 01 06:04:24.148353 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:24.148217 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-695f5e6f-5886d" event={"ID":"a7a48e16-9f97-4c4d-888e-9e5434f7233d","Type":"ContainerStarted","Data":"a8ce76f8376d056a807d7b0f87001810de5265ea0c2d7fc0aec80cc9b62f8790"} Nov 01 06:04:24.646949 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:24.646876 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-695f5e6f-5886d" podStartSLOduration=2.646856102 podStartE2EDuration="2.646856102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:04:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:04:24.643032819 +0000 UTC m=+1143.298149989" watchObservedRunningTime="2026-11-01 06:04:24.646856102 +0000 UTC m=+1143.301973270" Nov 01 06:04:25.226448 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:25.225861 5040 generic.go:356] "Generic (PLEG): container finished" podID="a7a48e16-9f97-4c4d-888e-9e5434f7233d" containerID="5ca903bd862220e6f744b5fb89c73db0d6f5addc4d7c765b8dfb8112ab2ef4d6" exitCode=0 Nov 01 06:04:25.226448 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:25.226026 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-695f5e6f-5886d" event={"ID":"a7a48e16-9f97-4c4d-888e-9e5434f7233d","Type":"ContainerDied","Data":"5ca903bd862220e6f744b5fb89c73db0d6f5addc4d7c765b8dfb8112ab2ef4d6"} Nov 01 06:04:26.373877 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:26.373848 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-695f5e6f-5886d" Nov 01 06:04:26.531848 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:26.531814 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7a48e16-9f97-4c4d-888e-9e5434f7233d-kube-api-access-nnm22\" (UniqueName: \"kubernetes.io/projected/a7a48e16-9f97-4c4d-888e-9e5434f7233d-kube-api-access-nnm22\") pod \"a7a48e16-9f97-4c4d-888e-9e5434f7233d\" (UID: \"a7a48e16-9f97-4c4d-888e-9e5434f7233d\") " Nov 01 06:04:26.532051 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:26.531860 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7a48e16-9f97-4c4d-888e-9e5434f7233d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7a48e16-9f97-4c4d-888e-9e5434f7233d-maas-api-xks-ca\") pod \"a7a48e16-9f97-4c4d-888e-9e5434f7233d\" (UID: \"a7a48e16-9f97-4c4d-888e-9e5434f7233d\") " Nov 01 06:04:26.532051 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:26.531920 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7a48e16-9f97-4c4d-888e-9e5434f7233d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7a48e16-9f97-4c4d-888e-9e5434f7233d-maas-api-openshift-ca\") pod \"a7a48e16-9f97-4c4d-888e-9e5434f7233d\" (UID: \"a7a48e16-9f97-4c4d-888e-9e5434f7233d\") " Nov 01 06:04:26.532366 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:26.532336 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7a48e16-9f97-4c4d-888e-9e5434f7233d-maas-api-openshift-ca" pod "a7a48e16-9f97-4c4d-888e-9e5434f7233d" (UID: "a7a48e16-9f97-4c4d-888e-9e5434f7233d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:04:26.533978 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:26.533950 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7a48e16-9f97-4c4d-888e-9e5434f7233d-maas-api-xks-ca" pod "a7a48e16-9f97-4c4d-888e-9e5434f7233d" (UID: "a7a48e16-9f97-4c4d-888e-9e5434f7233d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:04:26.534099 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:26.534016 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a48e16-9f97-4c4d-888e-9e5434f7233d-kube-api-access-nnm22" pod "a7a48e16-9f97-4c4d-888e-9e5434f7233d" (UID: "a7a48e16-9f97-4c4d-888e-9e5434f7233d"). InnerVolumeSpecName "kube-api-access-nnm22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:04:26.633474 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:26.633433 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnm22\" (UniqueName: \"kubernetes.io/projected/a7a48e16-9f97-4c4d-888e-9e5434f7233d-kube-api-access-nnm22\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:26.633474 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:26.633469 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7a48e16-9f97-4c4d-888e-9e5434f7233d-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:26.633684 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:26.633506 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7a48e16-9f97-4c4d-888e-9e5434f7233d-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:27.242675 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:27.242645 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-695f5e6f-5886d" Nov 01 06:04:27.242880 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:27.242687 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-695f5e6f-5886d" event={"ID":"a7a48e16-9f97-4c4d-888e-9e5434f7233d","Type":"ContainerDied","Data":"a8ce76f8376d056a807d7b0f87001810de5265ea0c2d7fc0aec80cc9b62f8790"} Nov 01 06:04:27.242880 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:27.242746 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8ce76f8376d056a807d7b0f87001810de5265ea0c2d7fc0aec80cc9b62f8790" Nov 01 06:04:27.954142 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:27.954104 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-mgnkl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 06:04:27.954615 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:27.954176 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" podUID="baa43c0f-ec86-4262-9bdc-773e9b11bdbb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 06:04:29.280826 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.280555 5040 generic.go:356] "Generic (PLEG): container finished" podID="67d2bd89-b64a-4f9e-ba55-d9cb9caf1907" containerID="a83a4164778bda232058de40b85b77529ada83cb1cb1bba8ca179f21b9c16a97" exitCode=0 Nov 01 06:04:29.280826 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.280632 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" event={"ID":"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907","Type":"ContainerDied","Data":"a83a4164778bda232058de40b85b77529ada83cb1cb1bba8ca179f21b9c16a97"} Nov 01 06:04:29.284518 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.284462 5040 generic.go:356] "Generic (PLEG): container finished" podID="0aeeb98d-3738-4319-8531-dba6f0705696" containerID="9799a124214b067052467c5b105639c3ca2461227b57c8eeba925d26f3566501" exitCode=0 Nov 01 06:04:29.284762 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.284525 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" event={"ID":"0aeeb98d-3738-4319-8531-dba6f0705696","Type":"ContainerDied","Data":"9799a124214b067052467c5b105639c3ca2461227b57c8eeba925d26f3566501"} Nov 01 06:04:29.286981 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.286960 5040 generic.go:356] "Generic (PLEG): container finished" podID="baa43c0f-ec86-4262-9bdc-773e9b11bdbb" containerID="70fd85aa5cc8403e134941f7c20cabed7f6e1c089e2995c2dcaf4da71f9eda82" exitCode=0 Nov 01 06:04:29.287100 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.287080 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" event={"ID":"baa43c0f-ec86-4262-9bdc-773e9b11bdbb","Type":"ContainerDied","Data":"70fd85aa5cc8403e134941f7c20cabed7f6e1c089e2995c2dcaf4da71f9eda82"} Nov 01 06:04:29.287175 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.287131 5040 scope.go:122] "RemoveContainer" containerID="2ef15770562fae9547d3fdcef5b63227b8bef2b9218426060ac98e29e6f41618" Nov 01 06:04:29.289400 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.289373 5040 generic.go:356] "Generic (PLEG): container finished" podID="bba017ed-c86e-4442-a083-2a828ad624b1" containerID="2716189ca52914194cf6269d27c0fd8bb78d3d60b8c9d57a9fcc0fff13eee741" exitCode=0 Nov 01 06:04:29.289521 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.289421 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" event={"ID":"bba017ed-c86e-4442-a083-2a828ad624b1","Type":"ContainerDied","Data":"2716189ca52914194cf6269d27c0fd8bb78d3d60b8c9d57a9fcc0fff13eee741"} Nov 01 06:04:29.367615 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.367592 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" Nov 01 06:04:29.393660 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.393547 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" Nov 01 06:04:29.445889 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.445864 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" Nov 01 06:04:29.450241 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.450221 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 06:04:29.464652 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.464631 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aeeb98d-3738-4319-8531-dba6f0705696-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aeeb98d-3738-4319-8531-dba6f0705696-credential-socket\") pod \"0aeeb98d-3738-4319-8531-dba6f0705696\" (UID: \"0aeeb98d-3738-4319-8531-dba6f0705696\") " Nov 01 06:04:29.464747 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.464709 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0aeeb98d-3738-4319-8531-dba6f0705696-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aeeb98d-3738-4319-8531-dba6f0705696-istio-podinfo\") pod \"0aeeb98d-3738-4319-8531-dba6f0705696\" (UID: \"0aeeb98d-3738-4319-8531-dba6f0705696\") " Nov 01 06:04:29.464800 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.464765 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aeeb98d-3738-4319-8531-dba6f0705696-kube-api-access-l7jgd\" (UniqueName: \"kubernetes.io/projected/0aeeb98d-3738-4319-8531-dba6f0705696-kube-api-access-l7jgd\") pod \"0aeeb98d-3738-4319-8531-dba6f0705696\" (UID: \"0aeeb98d-3738-4319-8531-dba6f0705696\") " Nov 01 06:04:29.464800 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.464791 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aeeb98d-3738-4319-8531-dba6f0705696-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aeeb98d-3738-4319-8531-dba6f0705696-istio-envoy\") pod \"0aeeb98d-3738-4319-8531-dba6f0705696\" (UID: \"0aeeb98d-3738-4319-8531-dba6f0705696\") " Nov 01 06:04:29.464884 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.464811 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aeeb98d-3738-4319-8531-dba6f0705696-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aeeb98d-3738-4319-8531-dba6f0705696-istiod-ca-cert\") pod \"0aeeb98d-3738-4319-8531-dba6f0705696\" (UID: \"0aeeb98d-3738-4319-8531-dba6f0705696\") " Nov 01 06:04:29.464884 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.464830 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aeeb98d-3738-4319-8531-dba6f0705696-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aeeb98d-3738-4319-8531-dba6f0705696-workload-certs\") pod \"0aeeb98d-3738-4319-8531-dba6f0705696\" (UID: \"0aeeb98d-3738-4319-8531-dba6f0705696\") " Nov 01 06:04:29.464884 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.464872 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aeeb98d-3738-4319-8531-dba6f0705696-istio-token\" (UniqueName: \"kubernetes.io/projected/0aeeb98d-3738-4319-8531-dba6f0705696-istio-token\") pod \"0aeeb98d-3738-4319-8531-dba6f0705696\" (UID: \"0aeeb98d-3738-4319-8531-dba6f0705696\") " Nov 01 06:04:29.465040 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.464885 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aeeb98d-3738-4319-8531-dba6f0705696-credential-socket" pod "0aeeb98d-3738-4319-8531-dba6f0705696" (UID: "0aeeb98d-3738-4319-8531-dba6f0705696"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:29.465040 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.464908 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aeeb98d-3738-4319-8531-dba6f0705696-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aeeb98d-3738-4319-8531-dba6f0705696-workload-socket\") pod \"0aeeb98d-3738-4319-8531-dba6f0705696\" (UID: \"0aeeb98d-3738-4319-8531-dba6f0705696\") " Nov 01 06:04:29.465040 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.464955 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aeeb98d-3738-4319-8531-dba6f0705696-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aeeb98d-3738-4319-8531-dba6f0705696-istio-data\") pod \"0aeeb98d-3738-4319-8531-dba6f0705696\" (UID: \"0aeeb98d-3738-4319-8531-dba6f0705696\") " Nov 01 06:04:29.465567 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.465259 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aeeb98d-3738-4319-8531-dba6f0705696-workload-certs" pod "0aeeb98d-3738-4319-8531-dba6f0705696" (UID: "0aeeb98d-3738-4319-8531-dba6f0705696"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:29.465567 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.465293 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aeeb98d-3738-4319-8531-dba6f0705696-istiod-ca-cert" pod "0aeeb98d-3738-4319-8531-dba6f0705696" (UID: "0aeeb98d-3738-4319-8531-dba6f0705696"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:04:29.465567 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.465467 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aeeb98d-3738-4319-8531-dba6f0705696-workload-socket" pod "0aeeb98d-3738-4319-8531-dba6f0705696" (UID: "0aeeb98d-3738-4319-8531-dba6f0705696"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:29.465567 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.465503 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aeeb98d-3738-4319-8531-dba6f0705696-istio-data" pod "0aeeb98d-3738-4319-8531-dba6f0705696" (UID: "0aeeb98d-3738-4319-8531-dba6f0705696"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:29.465567 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.465538 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aeeb98d-3738-4319-8531-dba6f0705696-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:29.465567 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.465559 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aeeb98d-3738-4319-8531-dba6f0705696-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:29.465950 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.465574 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aeeb98d-3738-4319-8531-dba6f0705696-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:29.467097 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.467075 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aeeb98d-3738-4319-8531-dba6f0705696-istio-token" pod "0aeeb98d-3738-4319-8531-dba6f0705696" (UID: "0aeeb98d-3738-4319-8531-dba6f0705696"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:04:29.467225 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.467190 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aeeb98d-3738-4319-8531-dba6f0705696-istio-envoy" pod "0aeeb98d-3738-4319-8531-dba6f0705696" (UID: "0aeeb98d-3738-4319-8531-dba6f0705696"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:29.467225 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.467199 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0aeeb98d-3738-4319-8531-dba6f0705696-istio-podinfo" pod "0aeeb98d-3738-4319-8531-dba6f0705696" (UID: "0aeeb98d-3738-4319-8531-dba6f0705696"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:04:29.467311 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.467282 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aeeb98d-3738-4319-8531-dba6f0705696-kube-api-access-l7jgd" pod "0aeeb98d-3738-4319-8531-dba6f0705696" (UID: "0aeeb98d-3738-4319-8531-dba6f0705696"). InnerVolumeSpecName "kube-api-access-l7jgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:04:29.566122 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.566032 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-istio-envoy\") pod \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\" (UID: \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\") " Nov 01 06:04:29.566122 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.566088 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bba017ed-c86e-4442-a083-2a828ad624b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba017ed-c86e-4442-a083-2a828ad624b1-workload-socket\") pod \"bba017ed-c86e-4442-a083-2a828ad624b1\" (UID: \"bba017ed-c86e-4442-a083-2a828ad624b1\") " Nov 01 06:04:29.566371 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.566130 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bba017ed-c86e-4442-a083-2a828ad624b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba017ed-c86e-4442-a083-2a828ad624b1-istiod-ca-cert\") pod \"bba017ed-c86e-4442-a083-2a828ad624b1\" (UID: \"bba017ed-c86e-4442-a083-2a828ad624b1\") " Nov 01 06:04:29.566371 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.566178 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bba017ed-c86e-4442-a083-2a828ad624b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba017ed-c86e-4442-a083-2a828ad624b1-istio-podinfo\") pod \"bba017ed-c86e-4442-a083-2a828ad624b1\" (UID: \"bba017ed-c86e-4442-a083-2a828ad624b1\") " Nov 01 06:04:29.566371 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.566218 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bba017ed-c86e-4442-a083-2a828ad624b1-kube-api-access-ns672\" (UniqueName: \"kubernetes.io/projected/bba017ed-c86e-4442-a083-2a828ad624b1-kube-api-access-ns672\") pod \"bba017ed-c86e-4442-a083-2a828ad624b1\" (UID: \"bba017ed-c86e-4442-a083-2a828ad624b1\") " Nov 01 06:04:29.566558 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.566385 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-workload-certs\") pod \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\" (UID: \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\") " Nov 01 06:04:29.566558 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.566424 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bba017ed-c86e-4442-a083-2a828ad624b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba017ed-c86e-4442-a083-2a828ad624b1-istio-envoy\") pod \"bba017ed-c86e-4442-a083-2a828ad624b1\" (UID: \"bba017ed-c86e-4442-a083-2a828ad624b1\") " Nov 01 06:04:29.566558 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.566455 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bba017ed-c86e-4442-a083-2a828ad624b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba017ed-c86e-4442-a083-2a828ad624b1-istio-data\") pod \"bba017ed-c86e-4442-a083-2a828ad624b1\" (UID: \"bba017ed-c86e-4442-a083-2a828ad624b1\") " Nov 01 06:04:29.566558 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.566504 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-credential-socket\") pod \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\" (UID: \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\") " Nov 01 06:04:29.566558 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.566534 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bba017ed-c86e-4442-a083-2a828ad624b1-istio-token\" (UniqueName: \"kubernetes.io/projected/bba017ed-c86e-4442-a083-2a828ad624b1-istio-token\") pod \"bba017ed-c86e-4442-a083-2a828ad624b1\" (UID: \"bba017ed-c86e-4442-a083-2a828ad624b1\") " Nov 01 06:04:29.566809 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.566565 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-istiod-ca-cert\") pod \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\" (UID: \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\") " Nov 01 06:04:29.566809 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.566603 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-workload-socket\") pod \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\" (UID: \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\") " Nov 01 06:04:29.566809 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.566641 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-istio-token\" (UniqueName: \"kubernetes.io/projected/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-istio-token\") pod \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\" (UID: \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\") " Nov 01 06:04:29.566809 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.566671 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-istio-podinfo\") pod \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\" (UID: \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\") " Nov 01 06:04:29.566809 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.566704 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-istio-token\" (UniqueName: \"kubernetes.io/projected/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-istio-token\") pod \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\" (UID: \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\") " Nov 01 06:04:29.566809 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.566735 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-istio-data\") pod \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\" (UID: \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\") " Nov 01 06:04:29.566809 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.566768 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-credential-socket\") pod \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\" (UID: \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\") " Nov 01 06:04:29.566809 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.566795 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-istio-data\") pod \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\" (UID: \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\") " Nov 01 06:04:29.567211 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.566813 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-workload-certs" pod "baa43c0f-ec86-4262-9bdc-773e9b11bdbb" (UID: "baa43c0f-ec86-4262-9bdc-773e9b11bdbb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:29.567211 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.566827 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-workload-certs\") pod \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\" (UID: \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\") " Nov 01 06:04:29.567211 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.566898 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bba017ed-c86e-4442-a083-2a828ad624b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba017ed-c86e-4442-a083-2a828ad624b1-credential-socket\") pod \"bba017ed-c86e-4442-a083-2a828ad624b1\" (UID: \"bba017ed-c86e-4442-a083-2a828ad624b1\") " Nov 01 06:04:29.567211 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.566941 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-istio-podinfo\") pod \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\" (UID: \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\") " Nov 01 06:04:29.567211 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.566965 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bba017ed-c86e-4442-a083-2a828ad624b1-istiod-ca-cert" pod "bba017ed-c86e-4442-a083-2a828ad624b1" (UID: "bba017ed-c86e-4442-a083-2a828ad624b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:04:29.567211 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.566979 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-workload-socket\") pod \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\" (UID: \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\") " Nov 01 06:04:29.567211 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.567077 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-credential-socket" pod "67d2bd89-b64a-4f9e-ba55-d9cb9caf1907" (UID: "67d2bd89-b64a-4f9e-ba55-d9cb9caf1907"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:29.567211 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.567100 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-workload-certs" pod "67d2bd89-b64a-4f9e-ba55-d9cb9caf1907" (UID: "67d2bd89-b64a-4f9e-ba55-d9cb9caf1907"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:29.567673 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.567341 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-istio-envoy\") pod \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\" (UID: \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\") " Nov 01 06:04:29.567673 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.567387 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bba017ed-c86e-4442-a083-2a828ad624b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba017ed-c86e-4442-a083-2a828ad624b1-workload-certs\") pod \"bba017ed-c86e-4442-a083-2a828ad624b1\" (UID: \"bba017ed-c86e-4442-a083-2a828ad624b1\") " Nov 01 06:04:29.567673 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.567425 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-kube-api-access-jbz7l\" (UniqueName: \"kubernetes.io/projected/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-kube-api-access-jbz7l\") pod \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\" (UID: \"baa43c0f-ec86-4262-9bdc-773e9b11bdbb\") " Nov 01 06:04:29.567673 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.567508 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-kube-api-access-5rblg\" (UniqueName: \"kubernetes.io/projected/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-kube-api-access-5rblg\") pod \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\" (UID: \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\") " Nov 01 06:04:29.567673 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.567543 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-istiod-ca-cert\") pod \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\" (UID: \"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907\") " Nov 01 06:04:29.568048 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.568025 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:29.568115 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.568054 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aeeb98d-3738-4319-8531-dba6f0705696-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:29.568115 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.568071 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba017ed-c86e-4442-a083-2a828ad624b1-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:29.568115 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.568085 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7jgd\" (UniqueName: \"kubernetes.io/projected/0aeeb98d-3738-4319-8531-dba6f0705696-kube-api-access-l7jgd\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:29.568115 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.568101 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aeeb98d-3738-4319-8531-dba6f0705696-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:29.568352 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.568116 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aeeb98d-3738-4319-8531-dba6f0705696-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:29.568352 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.568132 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aeeb98d-3738-4319-8531-dba6f0705696-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:29.568352 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.568146 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aeeb98d-3738-4319-8531-dba6f0705696-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:29.568352 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.568161 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:29.568352 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.568176 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:29.568676 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.568497 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba017ed-c86e-4442-a083-2a828ad624b1-workload-socket" pod "bba017ed-c86e-4442-a083-2a828ad624b1" (UID: "bba017ed-c86e-4442-a083-2a828ad624b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:29.568676 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.568608 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-istiod-ca-cert" pod "67d2bd89-b64a-4f9e-ba55-d9cb9caf1907" (UID: "67d2bd89-b64a-4f9e-ba55-d9cb9caf1907"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:04:29.568782 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.568730 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba017ed-c86e-4442-a083-2a828ad624b1-istio-data" pod "bba017ed-c86e-4442-a083-2a828ad624b1" (UID: "bba017ed-c86e-4442-a083-2a828ad624b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:29.568868 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.568816 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-istio-envoy" pod "baa43c0f-ec86-4262-9bdc-773e9b11bdbb" (UID: "baa43c0f-ec86-4262-9bdc-773e9b11bdbb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:29.568937 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.568884 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba017ed-c86e-4442-a083-2a828ad624b1-credential-socket" pod "bba017ed-c86e-4442-a083-2a828ad624b1" (UID: "bba017ed-c86e-4442-a083-2a828ad624b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:29.569059 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.569015 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-credential-socket" pod "baa43c0f-ec86-4262-9bdc-773e9b11bdbb" (UID: "baa43c0f-ec86-4262-9bdc-773e9b11bdbb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:29.569132 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.569113 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-istiod-ca-cert" pod "baa43c0f-ec86-4262-9bdc-773e9b11bdbb" (UID: "baa43c0f-ec86-4262-9bdc-773e9b11bdbb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:04:29.569355 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.569294 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-workload-socket" pod "67d2bd89-b64a-4f9e-ba55-d9cb9caf1907" (UID: "67d2bd89-b64a-4f9e-ba55-d9cb9caf1907"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:29.569732 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.569431 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba017ed-c86e-4442-a083-2a828ad624b1-workload-certs" pod "bba017ed-c86e-4442-a083-2a828ad624b1" (UID: "bba017ed-c86e-4442-a083-2a828ad624b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:29.569732 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.569459 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-istio-data" pod "baa43c0f-ec86-4262-9bdc-773e9b11bdbb" (UID: "baa43c0f-ec86-4262-9bdc-773e9b11bdbb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:29.569732 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.569560 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-workload-socket" pod "baa43c0f-ec86-4262-9bdc-773e9b11bdbb" (UID: "baa43c0f-ec86-4262-9bdc-773e9b11bdbb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:29.569950 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.569811 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-istio-data" pod "67d2bd89-b64a-4f9e-ba55-d9cb9caf1907" (UID: "67d2bd89-b64a-4f9e-ba55-d9cb9caf1907"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:29.570169 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.570117 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bba017ed-c86e-4442-a083-2a828ad624b1-kube-api-access-ns672" pod "bba017ed-c86e-4442-a083-2a828ad624b1" (UID: "bba017ed-c86e-4442-a083-2a828ad624b1"). InnerVolumeSpecName "kube-api-access-ns672". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:04:29.571581 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.571554 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-istio-token" pod "67d2bd89-b64a-4f9e-ba55-d9cb9caf1907" (UID: "67d2bd89-b64a-4f9e-ba55-d9cb9caf1907"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:04:29.571892 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.571813 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bba017ed-c86e-4442-a083-2a828ad624b1-istio-token" pod "bba017ed-c86e-4442-a083-2a828ad624b1" (UID: "bba017ed-c86e-4442-a083-2a828ad624b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:04:29.571892 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.571820 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba017ed-c86e-4442-a083-2a828ad624b1-istio-envoy" pod "bba017ed-c86e-4442-a083-2a828ad624b1" (UID: "bba017ed-c86e-4442-a083-2a828ad624b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:29.572432 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.572401 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-istio-token" pod "baa43c0f-ec86-4262-9bdc-773e9b11bdbb" (UID: "baa43c0f-ec86-4262-9bdc-773e9b11bdbb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:04:29.572565 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.572460 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-kube-api-access-5rblg" pod "67d2bd89-b64a-4f9e-ba55-d9cb9caf1907" (UID: "67d2bd89-b64a-4f9e-ba55-d9cb9caf1907"). InnerVolumeSpecName "kube-api-access-5rblg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:04:29.572710 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.572683 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-istio-podinfo" pod "baa43c0f-ec86-4262-9bdc-773e9b11bdbb" (UID: "baa43c0f-ec86-4262-9bdc-773e9b11bdbb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:04:29.572809 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.572793 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-istio-envoy" pod "67d2bd89-b64a-4f9e-ba55-d9cb9caf1907" (UID: "67d2bd89-b64a-4f9e-ba55-d9cb9caf1907"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:29.573569 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.573542 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bba017ed-c86e-4442-a083-2a828ad624b1-istio-podinfo" pod "bba017ed-c86e-4442-a083-2a828ad624b1" (UID: "bba017ed-c86e-4442-a083-2a828ad624b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:04:29.573672 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.573655 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-kube-api-access-jbz7l" pod "baa43c0f-ec86-4262-9bdc-773e9b11bdbb" (UID: "baa43c0f-ec86-4262-9bdc-773e9b11bdbb"). InnerVolumeSpecName "kube-api-access-jbz7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:04:29.573904 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.573878 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-istio-podinfo" pod "67d2bd89-b64a-4f9e-ba55-d9cb9caf1907" (UID: "67d2bd89-b64a-4f9e-ba55-d9cb9caf1907"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:04:29.616749 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.616714 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj"] Nov 01 06:04:29.617151 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.617139 5040 cpu_manager.go:414] "containerMap: removing container" podUID="0aeeb98d-3738-4319-8531-dba6f0705696" containerName="istio-proxy" Nov 01 06:04:29.617196 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.617152 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aeeb98d-3738-4319-8531-dba6f0705696" containerName="istio-proxy" Nov 01 06:04:29.617196 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.617168 5040 cpu_manager.go:414] "containerMap: removing container" podUID="67d2bd89-b64a-4f9e-ba55-d9cb9caf1907" containerName="istio-proxy" Nov 01 06:04:29.617196 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.617173 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67d2bd89-b64a-4f9e-ba55-d9cb9caf1907" containerName="istio-proxy" Nov 01 06:04:29.617196 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.617183 5040 cpu_manager.go:414] "containerMap: removing container" podUID="baa43c0f-ec86-4262-9bdc-773e9b11bdbb" containerName="istio-proxy" Nov 01 06:04:29.617196 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.617188 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baa43c0f-ec86-4262-9bdc-773e9b11bdbb" containerName="istio-proxy" Nov 01 06:04:29.617348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.617204 5040 cpu_manager.go:414] "containerMap: removing container" podUID="a7a48e16-9f97-4c4d-888e-9e5434f7233d" containerName="revoke-keys" Nov 01 06:04:29.617348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.617209 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7a48e16-9f97-4c4d-888e-9e5434f7233d" containerName="revoke-keys" Nov 01 06:04:29.617348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.617219 5040 cpu_manager.go:414] "containerMap: removing container" podUID="bba017ed-c86e-4442-a083-2a828ad624b1" containerName="istio-proxy" Nov 01 06:04:29.617348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.617225 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bba017ed-c86e-4442-a083-2a828ad624b1" containerName="istio-proxy" Nov 01 06:04:29.617348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.617242 5040 cpu_manager.go:414] "containerMap: removing container" podUID="baa43c0f-ec86-4262-9bdc-773e9b11bdbb" containerName="istio-proxy" Nov 01 06:04:29.617348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.617246 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baa43c0f-ec86-4262-9bdc-773e9b11bdbb" containerName="istio-proxy" Nov 01 06:04:29.617348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.617306 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="baa43c0f-ec86-4262-9bdc-773e9b11bdbb" containerName="istio-proxy" Nov 01 06:04:29.617348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.617315 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7a48e16-9f97-4c4d-888e-9e5434f7233d" containerName="revoke-keys" Nov 01 06:04:29.617348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.617322 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="bba017ed-c86e-4442-a083-2a828ad624b1" containerName="istio-proxy" Nov 01 06:04:29.617348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.617331 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aeeb98d-3738-4319-8531-dba6f0705696" containerName="istio-proxy" Nov 01 06:04:29.617348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.617338 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="67d2bd89-b64a-4f9e-ba55-d9cb9caf1907" containerName="istio-proxy" Nov 01 06:04:29.622055 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.622032 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" Nov 01 06:04:29.670306 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.670274 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba017ed-c86e-4442-a083-2a828ad624b1-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:29.670306 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.670305 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba017ed-c86e-4442-a083-2a828ad624b1-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:29.670589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.670319 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba017ed-c86e-4442-a083-2a828ad624b1-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:29.670589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.670332 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:29.670589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.670347 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:29.670589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.670359 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:29.670589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.670371 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:29.670589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.670384 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:29.670589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.670395 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:29.670589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.670407 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:29.670589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.670421 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:29.670589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.670433 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba017ed-c86e-4442-a083-2a828ad624b1-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:29.670589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.670445 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:29.670589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.670457 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:29.670589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.670468 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:29.670589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.670497 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba017ed-c86e-4442-a083-2a828ad624b1-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:29.670589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.670510 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbz7l\" (UniqueName: \"kubernetes.io/projected/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-kube-api-access-jbz7l\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:29.670589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.670523 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rblg\" (UniqueName: \"kubernetes.io/projected/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-kube-api-access-5rblg\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:29.670589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.670537 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:29.670589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.670551 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baa43c0f-ec86-4262-9bdc-773e9b11bdbb-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:29.670589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.670567 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba017ed-c86e-4442-a083-2a828ad624b1-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:29.670589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.670580 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba017ed-c86e-4442-a083-2a828ad624b1-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:29.670589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.670595 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns672\" (UniqueName: \"kubernetes.io/projected/bba017ed-c86e-4442-a083-2a828ad624b1-kube-api-access-ns672\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:29.675465 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.675429 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-14a6e4\"/\"distinct-w2-14a6e4-kserve-self-signed-certs\"" Nov 01 06:04:29.718639 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.718608 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj"] Nov 01 06:04:29.772080 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.772042 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-tls-certs\") pod \"distinct-w2-14a6e4-kserve-85b55cc944-7wbnj\" (UID: \"75afe340-0b9c-4b4c-89ab-20d761e6b4f1\") " pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" Nov 01 06:04:29.772246 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.772104 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chmgt\" (UniqueName: \"kubernetes.io/projected/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-kube-api-access-chmgt\") pod \"distinct-w2-14a6e4-kserve-85b55cc944-7wbnj\" (UID: \"75afe340-0b9c-4b4c-89ab-20d761e6b4f1\") " pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" Nov 01 06:04:29.772246 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.772194 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-tmp-dir\") pod \"distinct-w2-14a6e4-kserve-85b55cc944-7wbnj\" (UID: \"75afe340-0b9c-4b4c-89ab-20d761e6b4f1\") " pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" Nov 01 06:04:29.772345 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.772255 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-home\") pod \"distinct-w2-14a6e4-kserve-85b55cc944-7wbnj\" (UID: \"75afe340-0b9c-4b4c-89ab-20d761e6b4f1\") " pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" Nov 01 06:04:29.772345 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.772330 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-dshm\") pod \"distinct-w2-14a6e4-kserve-85b55cc944-7wbnj\" (UID: \"75afe340-0b9c-4b4c-89ab-20d761e6b4f1\") " pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" Nov 01 06:04:29.772427 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.772364 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-model-cache\") pod \"distinct-w2-14a6e4-kserve-85b55cc944-7wbnj\" (UID: \"75afe340-0b9c-4b4c-89ab-20d761e6b4f1\") " pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" Nov 01 06:04:29.882976 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.874047 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-tls-certs\") pod \"distinct-w2-14a6e4-kserve-85b55cc944-7wbnj\" (UID: \"75afe340-0b9c-4b4c-89ab-20d761e6b4f1\") " pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" Nov 01 06:04:29.882976 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.874144 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chmgt\" (UniqueName: \"kubernetes.io/projected/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-kube-api-access-chmgt\") pod \"distinct-w2-14a6e4-kserve-85b55cc944-7wbnj\" (UID: \"75afe340-0b9c-4b4c-89ab-20d761e6b4f1\") " pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" Nov 01 06:04:29.882976 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.874199 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-tmp-dir\") pod \"distinct-w2-14a6e4-kserve-85b55cc944-7wbnj\" (UID: \"75afe340-0b9c-4b4c-89ab-20d761e6b4f1\") " pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" Nov 01 06:04:29.882976 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.874279 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-home\") pod \"distinct-w2-14a6e4-kserve-85b55cc944-7wbnj\" (UID: \"75afe340-0b9c-4b4c-89ab-20d761e6b4f1\") " pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" Nov 01 06:04:29.882976 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.874324 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-dshm\") pod \"distinct-w2-14a6e4-kserve-85b55cc944-7wbnj\" (UID: \"75afe340-0b9c-4b4c-89ab-20d761e6b4f1\") " pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" Nov 01 06:04:29.882976 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.874358 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-model-cache\") pod \"distinct-w2-14a6e4-kserve-85b55cc944-7wbnj\" (UID: \"75afe340-0b9c-4b4c-89ab-20d761e6b4f1\") " pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" Nov 01 06:04:29.882976 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.874918 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-model-cache\") pod \"distinct-w2-14a6e4-kserve-85b55cc944-7wbnj\" (UID: \"75afe340-0b9c-4b4c-89ab-20d761e6b4f1\") " pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" Nov 01 06:04:29.882976 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.875574 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-tmp-dir\") pod \"distinct-w2-14a6e4-kserve-85b55cc944-7wbnj\" (UID: \"75afe340-0b9c-4b4c-89ab-20d761e6b4f1\") " pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" Nov 01 06:04:29.882976 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.875822 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-home\") pod \"distinct-w2-14a6e4-kserve-85b55cc944-7wbnj\" (UID: \"75afe340-0b9c-4b4c-89ab-20d761e6b4f1\") " pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" Nov 01 06:04:29.882976 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.878686 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-tls-certs\") pod \"distinct-w2-14a6e4-kserve-85b55cc944-7wbnj\" (UID: \"75afe340-0b9c-4b4c-89ab-20d761e6b4f1\") " pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" Nov 01 06:04:29.882976 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.880547 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-dshm\") pod \"distinct-w2-14a6e4-kserve-85b55cc944-7wbnj\" (UID: \"75afe340-0b9c-4b4c-89ab-20d761e6b4f1\") " pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" Nov 01 06:04:29.938074 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:29.938036 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chmgt\" (UniqueName: \"kubernetes.io/projected/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-kube-api-access-chmgt\") pod \"distinct-w2-14a6e4-kserve-85b55cc944-7wbnj\" (UID: \"75afe340-0b9c-4b4c-89ab-20d761e6b4f1\") " pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" Nov 01 06:04:30.234249 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:30.234160 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" Nov 01 06:04:30.300800 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:30.300524 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" Nov 01 06:04:30.300800 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:30.300550 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h" event={"ID":"bba017ed-c86e-4442-a083-2a828ad624b1","Type":"ContainerDied","Data":"197233d60215c8b49367f04a5efe12aaffc91ace195d965d0e66fddfec52ff54"} Nov 01 06:04:30.300800 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:30.300670 5040 scope.go:122] "RemoveContainer" containerID="2716189ca52914194cf6269d27c0fd8bb78d3d60b8c9d57a9fcc0fff13eee741" Nov 01 06:04:30.303715 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:30.303359 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" event={"ID":"67d2bd89-b64a-4f9e-ba55-d9cb9caf1907","Type":"ContainerDied","Data":"ee880f82dc7fed54f77921561d5f23bc8d2994311a3c1749158313c9dd7b15a9"} Nov 01 06:04:30.303715 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:30.303565 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg" Nov 01 06:04:30.306346 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:30.305422 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" event={"ID":"0aeeb98d-3738-4319-8531-dba6f0705696","Type":"ContainerDied","Data":"e3cf79fd28a63e41c46508eb27de97b12f22891f97ee622572f4819332b7770f"} Nov 01 06:04:30.306346 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:30.305550 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf" Nov 01 06:04:30.309228 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:30.308914 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" event={"ID":"baa43c0f-ec86-4262-9bdc-773e9b11bdbb","Type":"ContainerDied","Data":"046f4bc58fd2d7f77a6d2972df9b4d96313dc5107cf863b5572d8d27d3b2555e"} Nov 01 06:04:30.309228 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:30.308935 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl" Nov 01 06:04:30.319145 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:30.319121 5040 scope.go:122] "RemoveContainer" containerID="a83a4164778bda232058de40b85b77529ada83cb1cb1bba8ca179f21b9c16a97" Nov 01 06:04:30.329078 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:30.329057 5040 scope.go:122] "RemoveContainer" containerID="9799a124214b067052467c5b105639c3ca2461227b57c8eeba925d26f3566501" Nov 01 06:04:30.340174 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:30.340157 5040 scope.go:122] "RemoveContainer" containerID="70fd85aa5cc8403e134941f7c20cabed7f6e1c089e2995c2dcaf4da71f9eda82" Nov 01 06:04:30.469898 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:30.469852 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg"] Nov 01 06:04:30.474939 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:30.474914 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-k99rg"] Nov 01 06:04:30.586441 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:30.586408 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl"] Nov 01 06:04:30.590918 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:30.590885 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mgnkl"] Nov 01 06:04:30.611498 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:30.611454 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj"] Nov 01 06:04:30.612346 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:04:30.612323 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75afe340_0b9c_4b4c_89ab_20d761e6b4f1.slice/crio-86d294025e93b037ed9348a4baf29761a468f90e797913038e1450aeba86f07e WatchSource:0}: Error finding container 86d294025e93b037ed9348a4baf29761a468f90e797913038e1450aeba86f07e: Status 404 returned error can't find the container with id 86d294025e93b037ed9348a4baf29761a468f90e797913038e1450aeba86f07e Nov 01 06:04:30.706520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:30.706459 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf"] Nov 01 06:04:30.710296 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:30.710268 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d5hgf"] Nov 01 06:04:30.818315 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:30.818279 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h"] Nov 01 06:04:30.822348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:30.822321 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tm67h"] Nov 01 06:04:31.317906 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:31.317869 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" event={"ID":"75afe340-0b9c-4b4c-89ab-20d761e6b4f1","Type":"ContainerStarted","Data":"dd00248788b40a58b0244afaa32709485526feaf81eb0fc7f6090c0bcaed951f"} Nov 01 06:04:31.318364 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:31.317914 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" event={"ID":"75afe340-0b9c-4b4c-89ab-20d761e6b4f1","Type":"ContainerStarted","Data":"86d294025e93b037ed9348a4baf29761a468f90e797913038e1450aeba86f07e"} Nov 01 06:04:31.439016 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:31.438963 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" podStartSLOduration=2.438948743 podStartE2EDuration="2.438948743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:04:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:04:31.435547926 +0000 UTC m=+1150.090665094" watchObservedRunningTime="2026-11-01 06:04:31.438948743 +0000 UTC m=+1150.094065939" Nov 01 06:04:31.515312 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:31.515281 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aeeb98d-3738-4319-8531-dba6f0705696" path="/var/lib/kubelet/pods/0aeeb98d-3738-4319-8531-dba6f0705696/volumes" Nov 01 06:04:31.515806 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:31.515784 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67d2bd89-b64a-4f9e-ba55-d9cb9caf1907" path="/var/lib/kubelet/pods/67d2bd89-b64a-4f9e-ba55-d9cb9caf1907/volumes" Nov 01 06:04:31.516223 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:31.516202 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baa43c0f-ec86-4262-9bdc-773e9b11bdbb" path="/var/lib/kubelet/pods/baa43c0f-ec86-4262-9bdc-773e9b11bdbb/volumes" Nov 01 06:04:31.517003 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:31.516984 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bba017ed-c86e-4442-a083-2a828ad624b1" path="/var/lib/kubelet/pods/bba017ed-c86e-4442-a083-2a828ad624b1/volumes" Nov 01 06:04:32.234604 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:32.234572 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" Nov 01 06:04:40.082091 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:04:40.081993 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:04:40.236594 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:40.236549 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" Nov 01 06:04:40.262932 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:40.262896 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" Nov 01 06:04:40.398180 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:40.398071 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" Nov 01 06:04:44.848342 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:44.848306 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h"] Nov 01 06:04:44.848870 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:44.848855 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="baa43c0f-ec86-4262-9bdc-773e9b11bdbb" containerName="istio-proxy" Nov 01 06:04:44.852475 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:44.852452 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:04:44.908181 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:44.908150 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-2d3f8d-openshift-default-dockercfg-zw2df\"" Nov 01 06:04:44.947625 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:44.947585 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0073efde-c913-424b-93e8-bda25a24a209-istio-data\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h\" (UID: \"0073efde-c913-424b-93e8-bda25a24a209\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:04:44.947832 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:44.947642 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0073efde-c913-424b-93e8-bda25a24a209-istio-podinfo\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h\" (UID: \"0073efde-c913-424b-93e8-bda25a24a209\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:04:44.947832 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:44.947737 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0073efde-c913-424b-93e8-bda25a24a209-credential-socket\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h\" (UID: \"0073efde-c913-424b-93e8-bda25a24a209\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:04:44.947832 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:44.947810 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0073efde-c913-424b-93e8-bda25a24a209-istio-token\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h\" (UID: \"0073efde-c913-424b-93e8-bda25a24a209\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:04:44.947989 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:44.947842 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xc7c\" (UniqueName: \"kubernetes.io/projected/0073efde-c913-424b-93e8-bda25a24a209-kube-api-access-6xc7c\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h\" (UID: \"0073efde-c913-424b-93e8-bda25a24a209\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:04:44.947989 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:44.947868 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0073efde-c913-424b-93e8-bda25a24a209-istiod-ca-cert\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h\" (UID: \"0073efde-c913-424b-93e8-bda25a24a209\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:04:44.947989 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:44.947967 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0073efde-c913-424b-93e8-bda25a24a209-workload-certs\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h\" (UID: \"0073efde-c913-424b-93e8-bda25a24a209\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:04:44.948143 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:44.947992 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0073efde-c913-424b-93e8-bda25a24a209-istio-envoy\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h\" (UID: \"0073efde-c913-424b-93e8-bda25a24a209\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:04:44.948143 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:44.948012 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0073efde-c913-424b-93e8-bda25a24a209-workload-socket\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h\" (UID: \"0073efde-c913-424b-93e8-bda25a24a209\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:04:44.963847 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:44.963810 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h"] Nov 01 06:04:45.049167 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:45.049125 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0073efde-c913-424b-93e8-bda25a24a209-workload-certs\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h\" (UID: \"0073efde-c913-424b-93e8-bda25a24a209\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:04:45.075087 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:45.075058 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0073efde-c913-424b-93e8-bda25a24a209-istio-envoy\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h\" (UID: \"0073efde-c913-424b-93e8-bda25a24a209\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:04:45.075358 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:45.075338 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0073efde-c913-424b-93e8-bda25a24a209-workload-socket\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h\" (UID: \"0073efde-c913-424b-93e8-bda25a24a209\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:04:45.075677 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:45.075654 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0073efde-c913-424b-93e8-bda25a24a209-istio-data\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h\" (UID: \"0073efde-c913-424b-93e8-bda25a24a209\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:04:45.075842 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:45.075825 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0073efde-c913-424b-93e8-bda25a24a209-istio-podinfo\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h\" (UID: \"0073efde-c913-424b-93e8-bda25a24a209\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:04:45.076017 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:45.075993 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0073efde-c913-424b-93e8-bda25a24a209-credential-socket\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h\" (UID: \"0073efde-c913-424b-93e8-bda25a24a209\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:04:45.076243 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:45.076226 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0073efde-c913-424b-93e8-bda25a24a209-istio-token\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h\" (UID: \"0073efde-c913-424b-93e8-bda25a24a209\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:04:45.076400 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:45.076385 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xc7c\" (UniqueName: \"kubernetes.io/projected/0073efde-c913-424b-93e8-bda25a24a209-kube-api-access-6xc7c\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h\" (UID: \"0073efde-c913-424b-93e8-bda25a24a209\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:04:45.076561 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:45.076546 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0073efde-c913-424b-93e8-bda25a24a209-istiod-ca-cert\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h\" (UID: \"0073efde-c913-424b-93e8-bda25a24a209\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:04:45.076665 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:45.074981 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0073efde-c913-424b-93e8-bda25a24a209-workload-certs\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h\" (UID: \"0073efde-c913-424b-93e8-bda25a24a209\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:04:45.077650 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:45.077622 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0073efde-c913-424b-93e8-bda25a24a209-istiod-ca-cert\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h\" (UID: \"0073efde-c913-424b-93e8-bda25a24a209\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:04:45.077966 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:45.077910 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0073efde-c913-424b-93e8-bda25a24a209-istio-data\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h\" (UID: \"0073efde-c913-424b-93e8-bda25a24a209\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:04:45.078059 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:45.076638 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0073efde-c913-424b-93e8-bda25a24a209-workload-socket\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h\" (UID: \"0073efde-c913-424b-93e8-bda25a24a209\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:04:45.078527 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:45.078509 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0073efde-c913-424b-93e8-bda25a24a209-credential-socket\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h\" (UID: \"0073efde-c913-424b-93e8-bda25a24a209\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:04:45.081537 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:45.081517 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0073efde-c913-424b-93e8-bda25a24a209-istio-envoy\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h\" (UID: \"0073efde-c913-424b-93e8-bda25a24a209\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:04:45.088208 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:45.088177 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0073efde-c913-424b-93e8-bda25a24a209-istio-podinfo\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h\" (UID: \"0073efde-c913-424b-93e8-bda25a24a209\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:04:45.140438 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:45.140357 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0073efde-c913-424b-93e8-bda25a24a209-istio-token\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h\" (UID: \"0073efde-c913-424b-93e8-bda25a24a209\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:04:45.140627 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:45.140446 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xc7c\" (UniqueName: \"kubernetes.io/projected/0073efde-c913-424b-93e8-bda25a24a209-kube-api-access-6xc7c\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h\" (UID: \"0073efde-c913-424b-93e8-bda25a24a209\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:04:45.166097 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:45.166052 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:04:45.598849 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:45.598823 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h"] Nov 01 06:04:45.599582 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:04:45.599560 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0073efde_c913_424b_93e8_bda25a24a209.slice/crio-9e0bd4ff4980a8b7851082fec60572a5e7537752b41b56a0f2448fe24d4b487f WatchSource:0}: Error finding container 9e0bd4ff4980a8b7851082fec60572a5e7537752b41b56a0f2448fe24d4b487f: Status 404 returned error can't find the container with id 9e0bd4ff4980a8b7851082fec60572a5e7537752b41b56a0f2448fe24d4b487f Nov 01 06:04:45.601998 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:45.601965 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:04:45.602058 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:45.602034 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:04:45.602105 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:45.602061 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:04:46.005419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:46.005376 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs"] Nov 01 06:04:46.009331 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:46.009311 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs" Nov 01 06:04:46.063501 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:46.062819 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"agc-cred-28460a5a0934\"" Nov 01 06:04:46.063501 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:46.063003 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"routing-overlay\"" Nov 01 06:04:46.063501 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:46.063107 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"openshift-service-ca.crt\"" Nov 01 06:04:46.063501 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:46.063156 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"payload-processing-external-model-plugins-e2e-extmod-b-3986bf\"" Nov 01 06:04:46.063501 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:46.063271 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"payload-processing-external-model-e2e--cf3b8510-dockercfg-ftjt9\"" Nov 01 06:04:46.063501 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:46.063272 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"kube-root-ca.crt\"" Nov 01 06:04:46.067237 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:46.067216 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs"] Nov 01 06:04:46.088291 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:46.088242 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0da63cdc-8890-49d3-b6f5-ba7396bdc309-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs\" (UID: \"0da63cdc-8890-49d3-b6f5-ba7396bdc309\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs" Nov 01 06:04:46.088423 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:46.088301 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbhmj\" (UniqueName: \"kubernetes.io/projected/0da63cdc-8890-49d3-b6f5-ba7396bdc309-kube-api-access-sbhmj\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs\" (UID: \"0da63cdc-8890-49d3-b6f5-ba7396bdc309\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs" Nov 01 06:04:46.088524 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:46.088503 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0da63cdc-8890-49d3-b6f5-ba7396bdc309-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs\" (UID: \"0da63cdc-8890-49d3-b6f5-ba7396bdc309\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs" Nov 01 06:04:46.088566 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:46.088557 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0da63cdc-8890-49d3-b6f5-ba7396bdc309-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs\" (UID: \"0da63cdc-8890-49d3-b6f5-ba7396bdc309\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs" Nov 01 06:04:46.189140 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:46.189107 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbhmj\" (UniqueName: \"kubernetes.io/projected/0da63cdc-8890-49d3-b6f5-ba7396bdc309-kube-api-access-sbhmj\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs\" (UID: \"0da63cdc-8890-49d3-b6f5-ba7396bdc309\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs" Nov 01 06:04:46.189337 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:46.189221 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0da63cdc-8890-49d3-b6f5-ba7396bdc309-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs\" (UID: \"0da63cdc-8890-49d3-b6f5-ba7396bdc309\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs" Nov 01 06:04:46.189337 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:46.189260 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0da63cdc-8890-49d3-b6f5-ba7396bdc309-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs\" (UID: \"0da63cdc-8890-49d3-b6f5-ba7396bdc309\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs" Nov 01 06:04:46.189337 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:46.189278 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0da63cdc-8890-49d3-b6f5-ba7396bdc309-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs\" (UID: \"0da63cdc-8890-49d3-b6f5-ba7396bdc309\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs" Nov 01 06:04:46.189883 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:46.189863 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0da63cdc-8890-49d3-b6f5-ba7396bdc309-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs\" (UID: \"0da63cdc-8890-49d3-b6f5-ba7396bdc309\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs" Nov 01 06:04:46.190065 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:46.190046 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0da63cdc-8890-49d3-b6f5-ba7396bdc309-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs\" (UID: \"0da63cdc-8890-49d3-b6f5-ba7396bdc309\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs" Nov 01 06:04:46.191783 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:46.191759 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0da63cdc-8890-49d3-b6f5-ba7396bdc309-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs\" (UID: \"0da63cdc-8890-49d3-b6f5-ba7396bdc309\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs" Nov 01 06:04:46.250643 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:46.250613 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbhmj\" (UniqueName: \"kubernetes.io/projected/0da63cdc-8890-49d3-b6f5-ba7396bdc309-kube-api-access-sbhmj\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs\" (UID: \"0da63cdc-8890-49d3-b6f5-ba7396bdc309\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs" Nov 01 06:04:46.323615 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:46.323532 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs" Nov 01 06:04:46.424809 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:46.424776 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" event={"ID":"0073efde-c913-424b-93e8-bda25a24a209","Type":"ContainerStarted","Data":"73a7065d2e4f82ade11bd5eba20661c60ba5233484bfa20430f66a700499a859"} Nov 01 06:04:46.425012 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:46.424845 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" event={"ID":"0073efde-c913-424b-93e8-bda25a24a209","Type":"ContainerStarted","Data":"9e0bd4ff4980a8b7851082fec60572a5e7537752b41b56a0f2448fe24d4b487f"} Nov 01 06:04:46.548791 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:46.548735 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" podStartSLOduration=2.548722168 podStartE2EDuration="2.548722168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:04:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:04:46.545636013 +0000 UTC m=+1165.200753179" watchObservedRunningTime="2026-11-01 06:04:46.548722168 +0000 UTC m=+1165.203839334" Nov 01 06:04:46.698702 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:46.698674 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs"] Nov 01 06:04:46.699640 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:04:46.699613 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da63cdc_8890_49d3_b6f5_ba7396bdc309.slice/crio-372729aec3be735097534bb75abb8e1feac6a48eb4077ee952476bc6bea5c02d WatchSource:0}: Error finding container 372729aec3be735097534bb75abb8e1feac6a48eb4077ee952476bc6bea5c02d: Status 404 returned error can't find the container with id 372729aec3be735097534bb75abb8e1feac6a48eb4077ee952476bc6bea5c02d Nov 01 06:04:47.166634 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:47.166602 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:04:47.171746 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:47.171724 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:04:47.432228 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:47.432147 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs" event={"ID":"0da63cdc-8890-49d3-b6f5-ba7396bdc309","Type":"ContainerStarted","Data":"372729aec3be735097534bb75abb8e1feac6a48eb4077ee952476bc6bea5c02d"} Nov 01 06:04:47.432541 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:47.432516 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:04:47.433472 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:47.433453 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:04:50.456672 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:50.456635 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs" event={"ID":"0da63cdc-8890-49d3-b6f5-ba7396bdc309","Type":"ContainerStarted","Data":"1493964669af1a9c31e3ae4158bf1a5f29ca785226db702c6f3d27932db35038"} Nov 01 06:04:50.457087 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:50.456775 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs" Nov 01 06:04:50.577923 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:50.577873 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs" podStartSLOduration=2.677923548 podStartE2EDuration="5.577859005s" totalImagesPullingTime="2.899935457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:04:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:04:50.576548295 +0000 UTC m=+1169.231665461" watchObservedRunningTime="2026-11-01 06:04:50.577859005 +0000 UTC m=+1169.232976171" Nov 01 06:04:54.057711 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:54.057670 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z"] Nov 01 06:04:54.063849 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:54.063823 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" Nov 01 06:04:54.179364 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:54.179320 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6aab7db-e2e7-40ef-bf44-6873237c494f-istio-data\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z\" (UID: \"d6aab7db-e2e7-40ef-bf44-6873237c494f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" Nov 01 06:04:54.179799 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:54.179762 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6aab7db-e2e7-40ef-bf44-6873237c494f-istio-token\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z\" (UID: \"d6aab7db-e2e7-40ef-bf44-6873237c494f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" Nov 01 06:04:54.179928 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:54.179844 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6aab7db-e2e7-40ef-bf44-6873237c494f-istio-envoy\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z\" (UID: \"d6aab7db-e2e7-40ef-bf44-6873237c494f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" Nov 01 06:04:54.179928 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:54.179887 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6aab7db-e2e7-40ef-bf44-6873237c494f-istio-podinfo\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z\" (UID: \"d6aab7db-e2e7-40ef-bf44-6873237c494f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" Nov 01 06:04:54.180355 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:54.180326 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6aab7db-e2e7-40ef-bf44-6873237c494f-workload-certs\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z\" (UID: \"d6aab7db-e2e7-40ef-bf44-6873237c494f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" Nov 01 06:04:54.180430 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:54.180383 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6aab7db-e2e7-40ef-bf44-6873237c494f-workload-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z\" (UID: \"d6aab7db-e2e7-40ef-bf44-6873237c494f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" Nov 01 06:04:54.180519 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:54.180422 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6aab7db-e2e7-40ef-bf44-6873237c494f-credential-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z\" (UID: \"d6aab7db-e2e7-40ef-bf44-6873237c494f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" Nov 01 06:04:54.180595 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:54.180553 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm4zp\" (UniqueName: \"kubernetes.io/projected/d6aab7db-e2e7-40ef-bf44-6873237c494f-kube-api-access-pm4zp\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z\" (UID: \"d6aab7db-e2e7-40ef-bf44-6873237c494f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" Nov 01 06:04:54.180654 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:54.180582 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6aab7db-e2e7-40ef-bf44-6873237c494f-istiod-ca-cert\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z\" (UID: \"d6aab7db-e2e7-40ef-bf44-6873237c494f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" Nov 01 06:04:54.271561 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:54.271529 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z"] Nov 01 06:04:54.281681 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:54.281652 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6aab7db-e2e7-40ef-bf44-6873237c494f-istio-envoy\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z\" (UID: \"d6aab7db-e2e7-40ef-bf44-6873237c494f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" Nov 01 06:04:54.281854 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:54.281693 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6aab7db-e2e7-40ef-bf44-6873237c494f-istio-podinfo\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z\" (UID: \"d6aab7db-e2e7-40ef-bf44-6873237c494f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" Nov 01 06:04:54.281854 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:54.281742 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6aab7db-e2e7-40ef-bf44-6873237c494f-workload-certs\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z\" (UID: \"d6aab7db-e2e7-40ef-bf44-6873237c494f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" Nov 01 06:04:54.281854 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:54.281770 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6aab7db-e2e7-40ef-bf44-6873237c494f-workload-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z\" (UID: \"d6aab7db-e2e7-40ef-bf44-6873237c494f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" Nov 01 06:04:54.281854 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:54.281792 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6aab7db-e2e7-40ef-bf44-6873237c494f-credential-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z\" (UID: \"d6aab7db-e2e7-40ef-bf44-6873237c494f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" Nov 01 06:04:54.282096 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:54.281868 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm4zp\" (UniqueName: \"kubernetes.io/projected/d6aab7db-e2e7-40ef-bf44-6873237c494f-kube-api-access-pm4zp\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z\" (UID: \"d6aab7db-e2e7-40ef-bf44-6873237c494f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" Nov 01 06:04:54.282096 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:54.281899 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6aab7db-e2e7-40ef-bf44-6873237c494f-istiod-ca-cert\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z\" (UID: \"d6aab7db-e2e7-40ef-bf44-6873237c494f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" Nov 01 06:04:54.282096 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:54.281935 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6aab7db-e2e7-40ef-bf44-6873237c494f-istio-data\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z\" (UID: \"d6aab7db-e2e7-40ef-bf44-6873237c494f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" Nov 01 06:04:54.282096 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:54.281951 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6aab7db-e2e7-40ef-bf44-6873237c494f-istio-token\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z\" (UID: \"d6aab7db-e2e7-40ef-bf44-6873237c494f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" Nov 01 06:04:54.282389 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:54.282360 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6aab7db-e2e7-40ef-bf44-6873237c494f-workload-certs\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z\" (UID: \"d6aab7db-e2e7-40ef-bf44-6873237c494f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" Nov 01 06:04:54.282597 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:54.282572 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6aab7db-e2e7-40ef-bf44-6873237c494f-istio-data\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z\" (UID: \"d6aab7db-e2e7-40ef-bf44-6873237c494f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" Nov 01 06:04:54.282694 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:54.282392 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6aab7db-e2e7-40ef-bf44-6873237c494f-workload-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z\" (UID: \"d6aab7db-e2e7-40ef-bf44-6873237c494f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" Nov 01 06:04:54.282694 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:54.282647 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6aab7db-e2e7-40ef-bf44-6873237c494f-credential-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z\" (UID: \"d6aab7db-e2e7-40ef-bf44-6873237c494f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" Nov 01 06:04:54.283432 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:54.283403 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6aab7db-e2e7-40ef-bf44-6873237c494f-istiod-ca-cert\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z\" (UID: \"d6aab7db-e2e7-40ef-bf44-6873237c494f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" Nov 01 06:04:54.284619 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:54.284592 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6aab7db-e2e7-40ef-bf44-6873237c494f-istio-podinfo\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z\" (UID: \"d6aab7db-e2e7-40ef-bf44-6873237c494f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" Nov 01 06:04:54.284727 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:54.284632 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6aab7db-e2e7-40ef-bf44-6873237c494f-istio-envoy\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z\" (UID: \"d6aab7db-e2e7-40ef-bf44-6873237c494f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" Nov 01 06:04:54.344084 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:54.344009 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6aab7db-e2e7-40ef-bf44-6873237c494f-istio-token\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z\" (UID: \"d6aab7db-e2e7-40ef-bf44-6873237c494f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" Nov 01 06:04:54.344229 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:54.344121 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm4zp\" (UniqueName: \"kubernetes.io/projected/d6aab7db-e2e7-40ef-bf44-6873237c494f-kube-api-access-pm4zp\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z\" (UID: \"d6aab7db-e2e7-40ef-bf44-6873237c494f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" Nov 01 06:04:54.378163 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:54.378132 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" Nov 01 06:04:54.814286 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:04:54.814254 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6aab7db_e2e7_40ef_bf44_6873237c494f.slice/crio-aab3c2bdc213d94b68459c4f1462c5daabc2af9deb8d002a4fc7afa94278f989 WatchSource:0}: Error finding container aab3c2bdc213d94b68459c4f1462c5daabc2af9deb8d002a4fc7afa94278f989: Status 404 returned error can't find the container with id aab3c2bdc213d94b68459c4f1462c5daabc2af9deb8d002a4fc7afa94278f989 Nov 01 06:04:54.814406 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:54.814345 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z"] Nov 01 06:04:54.816696 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:54.816662 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:04:54.816779 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:54.816726 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:04:54.816779 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:54.816754 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:04:55.494518 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:55.493561 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" event={"ID":"d6aab7db-e2e7-40ef-bf44-6873237c494f","Type":"ContainerStarted","Data":"864c5b50992c92f17b5c30b35e8c357b3e03c43378d7fc20601c29802627b0dd"} Nov 01 06:04:55.494518 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:55.493625 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" event={"ID":"d6aab7db-e2e7-40ef-bf44-6873237c494f","Type":"ContainerStarted","Data":"aab3c2bdc213d94b68459c4f1462c5daabc2af9deb8d002a4fc7afa94278f989"} Nov 01 06:04:55.686789 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:55.686726 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" podStartSLOduration=1.686712056 podStartE2EDuration="1.686712056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:04:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:04:55.682656372 +0000 UTC m=+1174.337773536" watchObservedRunningTime="2026-11-01 06:04:55.686712056 +0000 UTC m=+1174.341829260" Nov 01 06:04:56.386508 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:56.386308 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" Nov 01 06:04:56.397505 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:56.395749 5040 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 01 06:04:56.397505 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:56.395810 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" podUID="d6aab7db-e2e7-40ef-bf44-6873237c494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 01 06:04:57.379109 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:57.379068 5040 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 01 06:04:57.379648 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:57.379200 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" podUID="d6aab7db-e2e7-40ef-bf44-6873237c494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 01 06:04:58.378808 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:58.378770 5040 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 01 06:04:58.379012 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:58.378921 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" podUID="d6aab7db-e2e7-40ef-bf44-6873237c494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 01 06:04:59.378588 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:59.378549 5040 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 01 06:04:59.379101 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:04:59.378598 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" podUID="d6aab7db-e2e7-40ef-bf44-6873237c494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 01 06:05:00.379529 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:00.379462 5040 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 01 06:05:00.380095 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:00.380052 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" podUID="d6aab7db-e2e7-40ef-bf44-6873237c494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 01 06:05:01.379016 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:01.378980 5040 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 01 06:05:01.379218 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:01.379030 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" podUID="d6aab7db-e2e7-40ef-bf44-6873237c494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 01 06:05:01.466748 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:01.466721 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs" Nov 01 06:05:02.383374 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:02.383344 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" Nov 01 06:05:02.383837 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:02.383817 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" Nov 01 06:05:02.384704 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:02.384684 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" Nov 01 06:05:04.179636 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.179600 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj"] Nov 01 06:05:04.184443 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.184426 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" Nov 01 06:05:04.230046 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.230016 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd"] Nov 01 06:05:04.233876 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.233859 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" Nov 01 06:05:04.285683 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.285648 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj"] Nov 01 06:05:04.309035 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.308998 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/313135e2-fa8e-467d-ba47-91b11869b8f8-istiod-ca-cert\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj\" (UID: \"313135e2-fa8e-467d-ba47-91b11869b8f8\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" Nov 01 06:05:04.309035 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.309035 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/313135e2-fa8e-467d-ba47-91b11869b8f8-istio-token\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj\" (UID: \"313135e2-fa8e-467d-ba47-91b11869b8f8\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" Nov 01 06:05:04.309300 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.309054 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/313135e2-fa8e-467d-ba47-91b11869b8f8-istio-envoy\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj\" (UID: \"313135e2-fa8e-467d-ba47-91b11869b8f8\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" Nov 01 06:05:04.309300 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.309190 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/313135e2-fa8e-467d-ba47-91b11869b8f8-istio-podinfo\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj\" (UID: \"313135e2-fa8e-467d-ba47-91b11869b8f8\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" Nov 01 06:05:04.309409 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.309297 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/313135e2-fa8e-467d-ba47-91b11869b8f8-credential-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj\" (UID: \"313135e2-fa8e-467d-ba47-91b11869b8f8\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" Nov 01 06:05:04.309409 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.309377 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/313135e2-fa8e-467d-ba47-91b11869b8f8-workload-certs\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj\" (UID: \"313135e2-fa8e-467d-ba47-91b11869b8f8\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" Nov 01 06:05:04.309541 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.309447 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/313135e2-fa8e-467d-ba47-91b11869b8f8-workload-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj\" (UID: \"313135e2-fa8e-467d-ba47-91b11869b8f8\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" Nov 01 06:05:04.309541 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.309523 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mqj7\" (UniqueName: \"kubernetes.io/projected/313135e2-fa8e-467d-ba47-91b11869b8f8-kube-api-access-9mqj7\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj\" (UID: \"313135e2-fa8e-467d-ba47-91b11869b8f8\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" Nov 01 06:05:04.309631 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.309549 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/313135e2-fa8e-467d-ba47-91b11869b8f8-istio-data\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj\" (UID: \"313135e2-fa8e-467d-ba47-91b11869b8f8\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" Nov 01 06:05:04.339154 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.339116 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd"] Nov 01 06:05:04.411075 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.411040 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-workload-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd\" (UID: \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" Nov 01 06:05:04.411361 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.411335 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-istio-podinfo\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd\" (UID: \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" Nov 01 06:05:04.411499 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.411376 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-istio-token\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd\" (UID: \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" Nov 01 06:05:04.411499 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.411406 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/313135e2-fa8e-467d-ba47-91b11869b8f8-istiod-ca-cert\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj\" (UID: \"313135e2-fa8e-467d-ba47-91b11869b8f8\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" Nov 01 06:05:04.411499 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.411427 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/313135e2-fa8e-467d-ba47-91b11869b8f8-istio-token\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj\" (UID: \"313135e2-fa8e-467d-ba47-91b11869b8f8\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" Nov 01 06:05:04.411499 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.411443 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-istiod-ca-cert\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd\" (UID: \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" Nov 01 06:05:04.411499 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.411469 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/313135e2-fa8e-467d-ba47-91b11869b8f8-istio-envoy\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj\" (UID: \"313135e2-fa8e-467d-ba47-91b11869b8f8\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" Nov 01 06:05:04.411815 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.411538 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/313135e2-fa8e-467d-ba47-91b11869b8f8-istio-podinfo\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj\" (UID: \"313135e2-fa8e-467d-ba47-91b11869b8f8\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" Nov 01 06:05:04.411815 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.411598 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/313135e2-fa8e-467d-ba47-91b11869b8f8-credential-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj\" (UID: \"313135e2-fa8e-467d-ba47-91b11869b8f8\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" Nov 01 06:05:04.411815 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.411649 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-credential-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd\" (UID: \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" Nov 01 06:05:04.411815 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.411708 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/313135e2-fa8e-467d-ba47-91b11869b8f8-workload-certs\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj\" (UID: \"313135e2-fa8e-467d-ba47-91b11869b8f8\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" Nov 01 06:05:04.411815 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.411733 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwz2t\" (UniqueName: \"kubernetes.io/projected/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-kube-api-access-xwz2t\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd\" (UID: \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" Nov 01 06:05:04.411815 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.411757 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-istio-data\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd\" (UID: \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" Nov 01 06:05:04.411815 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.411794 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/313135e2-fa8e-467d-ba47-91b11869b8f8-workload-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj\" (UID: \"313135e2-fa8e-467d-ba47-91b11869b8f8\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" Nov 01 06:05:04.412179 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.411823 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-workload-certs\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd\" (UID: \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" Nov 01 06:05:04.412179 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.411858 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mqj7\" (UniqueName: \"kubernetes.io/projected/313135e2-fa8e-467d-ba47-91b11869b8f8-kube-api-access-9mqj7\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj\" (UID: \"313135e2-fa8e-467d-ba47-91b11869b8f8\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" Nov 01 06:05:04.412179 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.411898 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/313135e2-fa8e-467d-ba47-91b11869b8f8-istio-data\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj\" (UID: \"313135e2-fa8e-467d-ba47-91b11869b8f8\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" Nov 01 06:05:04.412179 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.411929 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-istio-envoy\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd\" (UID: \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" Nov 01 06:05:04.413235 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.413074 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/313135e2-fa8e-467d-ba47-91b11869b8f8-istiod-ca-cert\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj\" (UID: \"313135e2-fa8e-467d-ba47-91b11869b8f8\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" Nov 01 06:05:04.413610 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.413582 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/313135e2-fa8e-467d-ba47-91b11869b8f8-workload-certs\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj\" (UID: \"313135e2-fa8e-467d-ba47-91b11869b8f8\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" Nov 01 06:05:04.413749 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.413589 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/313135e2-fa8e-467d-ba47-91b11869b8f8-istio-data\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj\" (UID: \"313135e2-fa8e-467d-ba47-91b11869b8f8\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" Nov 01 06:05:04.413948 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.413922 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/313135e2-fa8e-467d-ba47-91b11869b8f8-credential-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj\" (UID: \"313135e2-fa8e-467d-ba47-91b11869b8f8\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" Nov 01 06:05:04.414189 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.414166 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/313135e2-fa8e-467d-ba47-91b11869b8f8-workload-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj\" (UID: \"313135e2-fa8e-467d-ba47-91b11869b8f8\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" Nov 01 06:05:04.416079 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.416057 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/313135e2-fa8e-467d-ba47-91b11869b8f8-istio-podinfo\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj\" (UID: \"313135e2-fa8e-467d-ba47-91b11869b8f8\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" Nov 01 06:05:04.416241 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.416226 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/313135e2-fa8e-467d-ba47-91b11869b8f8-istio-envoy\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj\" (UID: \"313135e2-fa8e-467d-ba47-91b11869b8f8\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" Nov 01 06:05:04.474871 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.474792 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/313135e2-fa8e-467d-ba47-91b11869b8f8-istio-token\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj\" (UID: \"313135e2-fa8e-467d-ba47-91b11869b8f8\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" Nov 01 06:05:04.474871 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.474845 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mqj7\" (UniqueName: \"kubernetes.io/projected/313135e2-fa8e-467d-ba47-91b11869b8f8-kube-api-access-9mqj7\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj\" (UID: \"313135e2-fa8e-467d-ba47-91b11869b8f8\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" Nov 01 06:05:04.497768 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.497737 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" Nov 01 06:05:04.512816 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.512790 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-credential-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd\" (UID: \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" Nov 01 06:05:04.512931 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.512845 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwz2t\" (UniqueName: \"kubernetes.io/projected/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-kube-api-access-xwz2t\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd\" (UID: \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" Nov 01 06:05:04.512931 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.512872 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-istio-data\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd\" (UID: \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" Nov 01 06:05:04.512931 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.512905 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-workload-certs\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd\" (UID: \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" Nov 01 06:05:04.513093 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.512950 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-istio-envoy\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd\" (UID: \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" Nov 01 06:05:04.513093 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.513002 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-workload-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd\" (UID: \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" Nov 01 06:05:04.513093 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.513044 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-istio-podinfo\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd\" (UID: \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" Nov 01 06:05:04.513093 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.513075 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-istio-token\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd\" (UID: \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" Nov 01 06:05:04.513292 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.513106 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-istiod-ca-cert\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd\" (UID: \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" Nov 01 06:05:04.513292 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.513250 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-istio-data\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd\" (UID: \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" Nov 01 06:05:04.513553 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.513527 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-workload-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd\" (UID: \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" Nov 01 06:05:04.513722 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.513573 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-workload-certs\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd\" (UID: \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" Nov 01 06:05:04.513844 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.513822 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-credential-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd\" (UID: \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" Nov 01 06:05:04.514030 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.513926 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-istiod-ca-cert\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd\" (UID: \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" Nov 01 06:05:04.515666 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.515639 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-istio-envoy\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd\" (UID: \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" Nov 01 06:05:04.515850 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.515830 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-istio-podinfo\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd\" (UID: \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" Nov 01 06:05:04.575130 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.575089 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwz2t\" (UniqueName: \"kubernetes.io/projected/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-kube-api-access-xwz2t\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd\" (UID: \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" Nov 01 06:05:04.576071 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.576042 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-istio-token\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd\" (UID: \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" Nov 01 06:05:04.847353 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.847312 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" Nov 01 06:05:04.928659 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.928613 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj"] Nov 01 06:05:04.928822 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:05:04.928791 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod313135e2_fa8e_467d_ba47_91b11869b8f8.slice/crio-4b1cb543e74187fae6b7bf8c0ecbaa6880f69719091e001446fcc4763575d93f WatchSource:0}: Error finding container 4b1cb543e74187fae6b7bf8c0ecbaa6880f69719091e001446fcc4763575d93f: Status 404 returned error can't find the container with id 4b1cb543e74187fae6b7bf8c0ecbaa6880f69719091e001446fcc4763575d93f Nov 01 06:05:04.931230 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.931201 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:05:04.931311 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.931266 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:05:04.931311 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:04.931297 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:05:05.163237 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:05.163159 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0a7600e-cls4g"] Nov 01 06:05:05.167142 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:05.167114 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a7600e-cls4g" Nov 01 06:05:05.220344 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:05.220312 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:05:05.220878 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:05.220426 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:05:05.220878 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:05.220465 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:05:05.220878 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:05.220516 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:05:05.224301 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:05.224276 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0a7600e-cls4g"] Nov 01 06:05:05.235958 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:05.235924 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd"] Nov 01 06:05:05.241374 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:05:05.241345 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a7e8e9_21c8_4d8b_b6a9_af407459207c.slice/crio-0fced4948536555ec45e6a063deb416a6d411fa88e756e494e1cc1698985ab32 WatchSource:0}: Error finding container 0fced4948536555ec45e6a063deb416a6d411fa88e756e494e1cc1698985ab32: Status 404 returned error can't find the container with id 0fced4948536555ec45e6a063deb416a6d411fa88e756e494e1cc1698985ab32 Nov 01 06:05:05.244630 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:05.244592 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:05:05.244738 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:05.244662 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:05:05.244738 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:05.244695 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:05:05.323108 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:05.323077 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31442774-8721-46d1-a4af-3fe60a925167-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0a7600e-cls4g\" (UID: \"31442774-8721-46d1-a4af-3fe60a925167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a7600e-cls4g" Nov 01 06:05:05.323220 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:05.323197 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31442774-8721-46d1-a4af-3fe60a925167-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0a7600e-cls4g\" (UID: \"31442774-8721-46d1-a4af-3fe60a925167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a7600e-cls4g" Nov 01 06:05:05.323282 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:05.323264 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z5d5\" (UniqueName: \"kubernetes.io/projected/31442774-8721-46d1-a4af-3fe60a925167-kube-api-access-5z5d5\") pod \"maas-api-revoke-sub-e0a7600e-cls4g\" (UID: \"31442774-8721-46d1-a4af-3fe60a925167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a7600e-cls4g" Nov 01 06:05:05.424674 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:05.424584 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31442774-8721-46d1-a4af-3fe60a925167-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0a7600e-cls4g\" (UID: \"31442774-8721-46d1-a4af-3fe60a925167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a7600e-cls4g" Nov 01 06:05:05.424674 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:05.424639 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z5d5\" (UniqueName: \"kubernetes.io/projected/31442774-8721-46d1-a4af-3fe60a925167-kube-api-access-5z5d5\") pod \"maas-api-revoke-sub-e0a7600e-cls4g\" (UID: \"31442774-8721-46d1-a4af-3fe60a925167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a7600e-cls4g" Nov 01 06:05:05.424933 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:05.424754 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31442774-8721-46d1-a4af-3fe60a925167-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0a7600e-cls4g\" (UID: \"31442774-8721-46d1-a4af-3fe60a925167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a7600e-cls4g" Nov 01 06:05:05.425436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:05.425415 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31442774-8721-46d1-a4af-3fe60a925167-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0a7600e-cls4g\" (UID: \"31442774-8721-46d1-a4af-3fe60a925167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a7600e-cls4g" Nov 01 06:05:05.427045 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:05.427023 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31442774-8721-46d1-a4af-3fe60a925167-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0a7600e-cls4g\" (UID: \"31442774-8721-46d1-a4af-3fe60a925167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a7600e-cls4g" Nov 01 06:05:05.486442 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:05.486400 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z5d5\" (UniqueName: \"kubernetes.io/projected/31442774-8721-46d1-a4af-3fe60a925167-kube-api-access-5z5d5\") pod \"maas-api-revoke-sub-e0a7600e-cls4g\" (UID: \"31442774-8721-46d1-a4af-3fe60a925167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a7600e-cls4g" Nov 01 06:05:05.583548 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:05.583511 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" event={"ID":"e2a7e8e9-21c8-4d8b-b6a9-af407459207c","Type":"ContainerStarted","Data":"b255129a5b1694db2b8b91d0482c25d9cad8797206c6d64107e2244823faf18c"} Nov 01 06:05:05.583743 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:05.583578 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" event={"ID":"e2a7e8e9-21c8-4d8b-b6a9-af407459207c","Type":"ContainerStarted","Data":"0fced4948536555ec45e6a063deb416a6d411fa88e756e494e1cc1698985ab32"} Nov 01 06:05:05.585357 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:05.585326 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" event={"ID":"313135e2-fa8e-467d-ba47-91b11869b8f8","Type":"ContainerStarted","Data":"538ada513fae35add7ab3a02b5d2b611650ce11d317fe17ebe9a20b2751d5dae"} Nov 01 06:05:05.585513 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:05.585369 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" event={"ID":"313135e2-fa8e-467d-ba47-91b11869b8f8","Type":"ContainerStarted","Data":"4b1cb543e74187fae6b7bf8c0ecbaa6880f69719091e001446fcc4763575d93f"} Nov 01 06:05:05.782090 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:05.782057 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a7600e-cls4g" Nov 01 06:05:05.850504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:05.847104 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" podStartSLOduration=1.8470822980000001 podStartE2EDuration="1.847082298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:05:05.723897768 +0000 UTC m=+1184.379014938" watchObservedRunningTime="2026-11-01 06:05:05.847082298 +0000 UTC m=+1184.502199466" Nov 01 06:05:05.863504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:05.853616 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" Nov 01 06:05:06.237592 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:06.219386 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" podStartSLOduration=2.2193636469999998 podStartE2EDuration="2.219363647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:05:05.852603702 +0000 UTC m=+1184.507720869" watchObservedRunningTime="2026-11-01 06:05:06.219363647 +0000 UTC m=+1184.874480815" Nov 01 06:05:06.246831 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:06.244154 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0a7600e-cls4g"] Nov 01 06:05:06.248871 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:05:06.248835 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31442774_8721_46d1_a4af_3fe60a925167.slice/crio-d81c14d5f7e1567f61dc1daa9fecf908ce441e7df479df0bb7b66b8efbb1131e WatchSource:0}: Error finding container d81c14d5f7e1567f61dc1daa9fecf908ce441e7df479df0bb7b66b8efbb1131e: Status 404 returned error can't find the container with id d81c14d5f7e1567f61dc1daa9fecf908ce441e7df479df0bb7b66b8efbb1131e Nov 01 06:05:06.505023 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:06.504938 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" Nov 01 06:05:06.531944 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:06.530614 5040 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 01 06:05:06.531944 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:06.530680 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" podUID="313135e2-fa8e-467d-ba47-91b11869b8f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 01 06:05:06.593273 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:06.593237 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a7600e-cls4g" event={"ID":"31442774-8721-46d1-a4af-3fe60a925167","Type":"ContainerStarted","Data":"d81c14d5f7e1567f61dc1daa9fecf908ce441e7df479df0bb7b66b8efbb1131e"} Nov 01 06:05:06.849858 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:06.849767 5040 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 01 06:05:06.849858 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:06.849831 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" podUID="e2a7e8e9-21c8-4d8b-b6a9-af407459207c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 01 06:05:07.505506 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:07.502911 5040 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 01 06:05:07.505506 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:07.502977 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" podUID="313135e2-fa8e-467d-ba47-91b11869b8f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 01 06:05:07.645447 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:07.645341 5040 generic.go:356] "Generic (PLEG): container finished" podID="31442774-8721-46d1-a4af-3fe60a925167" containerID="f3460af015e74a44070445d64b84d0c057f348e5762b4ee0fed9bdd903dd3d70" exitCode=0 Nov 01 06:05:07.645852 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:07.645520 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a7600e-cls4g" event={"ID":"31442774-8721-46d1-a4af-3fe60a925167","Type":"ContainerDied","Data":"f3460af015e74a44070445d64b84d0c057f348e5762b4ee0fed9bdd903dd3d70"} Nov 01 06:05:07.848054 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:07.848010 5040 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 01 06:05:07.848240 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:07.848065 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" podUID="e2a7e8e9-21c8-4d8b-b6a9-af407459207c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 01 06:05:08.502344 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:08.501871 5040 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 01 06:05:08.502344 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:08.501936 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" podUID="313135e2-fa8e-467d-ba47-91b11869b8f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 01 06:05:08.848194 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:08.847844 5040 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 01 06:05:08.848194 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:08.847901 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" podUID="e2a7e8e9-21c8-4d8b-b6a9-af407459207c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 01 06:05:08.952055 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:08.952023 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a7600e-cls4g" Nov 01 06:05:08.983759 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:08.983722 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31442774-8721-46d1-a4af-3fe60a925167-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31442774-8721-46d1-a4af-3fe60a925167-maas-api-xks-ca\") pod \"31442774-8721-46d1-a4af-3fe60a925167\" (UID: \"31442774-8721-46d1-a4af-3fe60a925167\") " Nov 01 06:05:08.983920 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:08.983787 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31442774-8721-46d1-a4af-3fe60a925167-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31442774-8721-46d1-a4af-3fe60a925167-maas-api-openshift-ca\") pod \"31442774-8721-46d1-a4af-3fe60a925167\" (UID: \"31442774-8721-46d1-a4af-3fe60a925167\") " Nov 01 06:05:08.983920 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:08.983887 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31442774-8721-46d1-a4af-3fe60a925167-kube-api-access-5z5d5\" (UniqueName: \"kubernetes.io/projected/31442774-8721-46d1-a4af-3fe60a925167-kube-api-access-5z5d5\") pod \"31442774-8721-46d1-a4af-3fe60a925167\" (UID: \"31442774-8721-46d1-a4af-3fe60a925167\") " Nov 01 06:05:08.984593 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:08.984562 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31442774-8721-46d1-a4af-3fe60a925167-maas-api-openshift-ca" pod "31442774-8721-46d1-a4af-3fe60a925167" (UID: "31442774-8721-46d1-a4af-3fe60a925167"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:05:08.985903 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:08.985867 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31442774-8721-46d1-a4af-3fe60a925167-maas-api-xks-ca" pod "31442774-8721-46d1-a4af-3fe60a925167" (UID: "31442774-8721-46d1-a4af-3fe60a925167"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:05:08.986736 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:08.986692 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31442774-8721-46d1-a4af-3fe60a925167-kube-api-access-5z5d5" pod "31442774-8721-46d1-a4af-3fe60a925167" (UID: "31442774-8721-46d1-a4af-3fe60a925167"). InnerVolumeSpecName "kube-api-access-5z5d5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:05:09.085587 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:09.085549 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z5d5\" (UniqueName: \"kubernetes.io/projected/31442774-8721-46d1-a4af-3fe60a925167-kube-api-access-5z5d5\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:09.085587 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:09.085583 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31442774-8721-46d1-a4af-3fe60a925167-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:09.085587 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:09.085595 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31442774-8721-46d1-a4af-3fe60a925167-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:09.502398 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:09.501928 5040 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 01 06:05:09.502398 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:09.501996 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" podUID="313135e2-fa8e-467d-ba47-91b11869b8f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 01 06:05:09.669017 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:09.668985 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a7600e-cls4g" Nov 01 06:05:09.669017 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:09.668999 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a7600e-cls4g" event={"ID":"31442774-8721-46d1-a4af-3fe60a925167","Type":"ContainerDied","Data":"d81c14d5f7e1567f61dc1daa9fecf908ce441e7df479df0bb7b66b8efbb1131e"} Nov 01 06:05:09.669222 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:09.669069 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d81c14d5f7e1567f61dc1daa9fecf908ce441e7df479df0bb7b66b8efbb1131e" Nov 01 06:05:09.848637 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:09.848551 5040 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 01 06:05:09.848637 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:09.848629 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" podUID="e2a7e8e9-21c8-4d8b-b6a9-af407459207c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 01 06:05:09.976630 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:05:09.976582 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:05:10.501970 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:10.501927 5040 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 01 06:05:10.502318 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:10.502284 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" podUID="313135e2-fa8e-467d-ba47-91b11869b8f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 01 06:05:10.848038 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:10.847797 5040 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 01 06:05:10.848038 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:10.847859 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" podUID="e2a7e8e9-21c8-4d8b-b6a9-af407459207c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 01 06:05:11.501029 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:11.500985 5040 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 01 06:05:11.501554 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:11.501052 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" podUID="313135e2-fa8e-467d-ba47-91b11869b8f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 01 06:05:11.848586 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:11.848462 5040 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 01 06:05:11.848869 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:11.848827 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" podUID="e2a7e8e9-21c8-4d8b-b6a9-af407459207c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 01 06:05:12.918842 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:12.918806 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" Nov 01 06:05:12.971672 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:12.970812 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" Nov 01 06:05:12.971858 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:12.971723 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" Nov 01 06:05:13.475729 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:13.475684 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" Nov 01 06:05:13.476562 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:13.476541 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" Nov 01 06:05:13.477773 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:13.477750 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" Nov 01 06:05:20.707027 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:20.706984 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx"] Nov 01 06:05:20.707739 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:20.707716 5040 cpu_manager.go:414] "containerMap: removing container" podUID="31442774-8721-46d1-a4af-3fe60a925167" containerName="revoke-keys" Nov 01 06:05:20.707846 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:20.707740 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31442774-8721-46d1-a4af-3fe60a925167" containerName="revoke-keys" Nov 01 06:05:20.707919 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:20.707905 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="31442774-8721-46d1-a4af-3fe60a925167" containerName="revoke-keys" Nov 01 06:05:20.712582 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:20.712556 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" Nov 01 06:05:20.769247 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:20.769209 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx"] Nov 01 06:05:20.815386 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:20.815343 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-14a6e4\"/\"distinct-2-w2-14a6e4-kserve-self-signed-certs\"" Nov 01 06:05:20.832998 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:20.832957 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adbac784-4ded-4ca3-b985-3e59983fc637-home\") pod \"distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx\" (UID: \"adbac784-4ded-4ca3-b985-3e59983fc637\") " pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" Nov 01 06:05:20.833162 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:20.833013 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adbac784-4ded-4ca3-b985-3e59983fc637-model-cache\") pod \"distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx\" (UID: \"adbac784-4ded-4ca3-b985-3e59983fc637\") " pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" Nov 01 06:05:20.833162 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:20.833113 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adbac784-4ded-4ca3-b985-3e59983fc637-tls-certs\") pod \"distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx\" (UID: \"adbac784-4ded-4ca3-b985-3e59983fc637\") " pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" Nov 01 06:05:20.833239 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:20.833174 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adbac784-4ded-4ca3-b985-3e59983fc637-tmp-dir\") pod \"distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx\" (UID: \"adbac784-4ded-4ca3-b985-3e59983fc637\") " pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" Nov 01 06:05:20.833290 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:20.833276 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adbac784-4ded-4ca3-b985-3e59983fc637-dshm\") pod \"distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx\" (UID: \"adbac784-4ded-4ca3-b985-3e59983fc637\") " pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" Nov 01 06:05:20.833335 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:20.833320 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j9c8\" (UniqueName: \"kubernetes.io/projected/adbac784-4ded-4ca3-b985-3e59983fc637-kube-api-access-6j9c8\") pod \"distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx\" (UID: \"adbac784-4ded-4ca3-b985-3e59983fc637\") " pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" Nov 01 06:05:20.929647 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:20.929613 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-637fa5fc-5g4z9"] Nov 01 06:05:20.933748 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:20.933720 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637fa5fc-5g4z9" Nov 01 06:05:20.934222 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:20.934200 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j9c8\" (UniqueName: \"kubernetes.io/projected/adbac784-4ded-4ca3-b985-3e59983fc637-kube-api-access-6j9c8\") pod \"distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx\" (UID: \"adbac784-4ded-4ca3-b985-3e59983fc637\") " pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" Nov 01 06:05:20.934344 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:20.934269 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adbac784-4ded-4ca3-b985-3e59983fc637-home\") pod \"distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx\" (UID: \"adbac784-4ded-4ca3-b985-3e59983fc637\") " pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" Nov 01 06:05:20.934344 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:20.934313 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adbac784-4ded-4ca3-b985-3e59983fc637-model-cache\") pod \"distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx\" (UID: \"adbac784-4ded-4ca3-b985-3e59983fc637\") " pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" Nov 01 06:05:20.934452 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:20.934350 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adbac784-4ded-4ca3-b985-3e59983fc637-tls-certs\") pod \"distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx\" (UID: \"adbac784-4ded-4ca3-b985-3e59983fc637\") " pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" Nov 01 06:05:20.934452 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:20.934400 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adbac784-4ded-4ca3-b985-3e59983fc637-tmp-dir\") pod \"distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx\" (UID: \"adbac784-4ded-4ca3-b985-3e59983fc637\") " pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" Nov 01 06:05:20.934625 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:20.934526 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adbac784-4ded-4ca3-b985-3e59983fc637-dshm\") pod \"distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx\" (UID: \"adbac784-4ded-4ca3-b985-3e59983fc637\") " pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" Nov 01 06:05:20.934793 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:20.934773 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adbac784-4ded-4ca3-b985-3e59983fc637-home\") pod \"distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx\" (UID: \"adbac784-4ded-4ca3-b985-3e59983fc637\") " pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" Nov 01 06:05:20.934875 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:20.934840 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adbac784-4ded-4ca3-b985-3e59983fc637-model-cache\") pod \"distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx\" (UID: \"adbac784-4ded-4ca3-b985-3e59983fc637\") " pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" Nov 01 06:05:20.934875 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:20.934868 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adbac784-4ded-4ca3-b985-3e59983fc637-tmp-dir\") pod \"distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx\" (UID: \"adbac784-4ded-4ca3-b985-3e59983fc637\") " pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" Nov 01 06:05:20.936907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:20.936876 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adbac784-4ded-4ca3-b985-3e59983fc637-dshm\") pod \"distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx\" (UID: \"adbac784-4ded-4ca3-b985-3e59983fc637\") " pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" Nov 01 06:05:20.937033 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:20.937013 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adbac784-4ded-4ca3-b985-3e59983fc637-tls-certs\") pod \"distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx\" (UID: \"adbac784-4ded-4ca3-b985-3e59983fc637\") " pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" Nov 01 06:05:20.998094 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:20.993967 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:05:20.998094 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:20.994044 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:05:20.998094 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:20.994105 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:05:20.998094 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:20.994188 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:05:20.999865 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:20.999834 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-637fa5fc-5g4z9"] Nov 01 06:05:21.001077 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:21.001001 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j9c8\" (UniqueName: \"kubernetes.io/projected/adbac784-4ded-4ca3-b985-3e59983fc637-kube-api-access-6j9c8\") pod \"distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx\" (UID: \"adbac784-4ded-4ca3-b985-3e59983fc637\") " pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" Nov 01 06:05:21.024896 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:21.024853 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" Nov 01 06:05:21.035734 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:21.035691 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e1dd328-b0f8-4ea0-9aab-93837559f559-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-637fa5fc-5g4z9\" (UID: \"3e1dd328-b0f8-4ea0-9aab-93837559f559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637fa5fc-5g4z9" Nov 01 06:05:21.035920 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:21.035795 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e1dd328-b0f8-4ea0-9aab-93837559f559-maas-api-xks-ca\") pod \"maas-api-revoke-sub-637fa5fc-5g4z9\" (UID: \"3e1dd328-b0f8-4ea0-9aab-93837559f559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637fa5fc-5g4z9" Nov 01 06:05:21.035990 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:21.035932 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbxn5\" (UniqueName: \"kubernetes.io/projected/3e1dd328-b0f8-4ea0-9aab-93837559f559-kube-api-access-wbxn5\") pod \"maas-api-revoke-sub-637fa5fc-5g4z9\" (UID: \"3e1dd328-b0f8-4ea0-9aab-93837559f559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637fa5fc-5g4z9" Nov 01 06:05:21.141894 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:21.139894 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e1dd328-b0f8-4ea0-9aab-93837559f559-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-637fa5fc-5g4z9\" (UID: \"3e1dd328-b0f8-4ea0-9aab-93837559f559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637fa5fc-5g4z9" Nov 01 06:05:21.141894 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:21.140010 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e1dd328-b0f8-4ea0-9aab-93837559f559-maas-api-xks-ca\") pod \"maas-api-revoke-sub-637fa5fc-5g4z9\" (UID: \"3e1dd328-b0f8-4ea0-9aab-93837559f559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637fa5fc-5g4z9" Nov 01 06:05:21.141894 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:21.140082 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbxn5\" (UniqueName: \"kubernetes.io/projected/3e1dd328-b0f8-4ea0-9aab-93837559f559-kube-api-access-wbxn5\") pod \"maas-api-revoke-sub-637fa5fc-5g4z9\" (UID: \"3e1dd328-b0f8-4ea0-9aab-93837559f559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637fa5fc-5g4z9" Nov 01 06:05:21.141894 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:21.141324 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e1dd328-b0f8-4ea0-9aab-93837559f559-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-637fa5fc-5g4z9\" (UID: \"3e1dd328-b0f8-4ea0-9aab-93837559f559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637fa5fc-5g4z9" Nov 01 06:05:21.146114 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:21.145892 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e1dd328-b0f8-4ea0-9aab-93837559f559-maas-api-xks-ca\") pod \"maas-api-revoke-sub-637fa5fc-5g4z9\" (UID: \"3e1dd328-b0f8-4ea0-9aab-93837559f559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637fa5fc-5g4z9" Nov 01 06:05:21.202391 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:21.202360 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbxn5\" (UniqueName: \"kubernetes.io/projected/3e1dd328-b0f8-4ea0-9aab-93837559f559-kube-api-access-wbxn5\") pod \"maas-api-revoke-sub-637fa5fc-5g4z9\" (UID: \"3e1dd328-b0f8-4ea0-9aab-93837559f559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637fa5fc-5g4z9" Nov 01 06:05:21.260664 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:21.260569 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637fa5fc-5g4z9" Nov 01 06:05:21.426162 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:05:21.426130 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podadbac784_4ded_4ca3_b985_3e59983fc637.slice/crio-a94537997b8356bb44c30a3838da4e7732486ca106d4d7fa3b05fe3d95e8a70e WatchSource:0}: Error finding container a94537997b8356bb44c30a3838da4e7732486ca106d4d7fa3b05fe3d95e8a70e: Status 404 returned error can't find the container with id a94537997b8356bb44c30a3838da4e7732486ca106d4d7fa3b05fe3d95e8a70e Nov 01 06:05:21.426304 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:21.426212 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx"] Nov 01 06:05:21.643738 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:21.643705 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-637fa5fc-5g4z9"] Nov 01 06:05:21.646821 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:05:21.646790 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e1dd328_b0f8_4ea0_9aab_93837559f559.slice/crio-947b2cb6f37d167f1ccd6547760e170d6497cc9bc270856bc76ae8ef9f10c488 WatchSource:0}: Error finding container 947b2cb6f37d167f1ccd6547760e170d6497cc9bc270856bc76ae8ef9f10c488: Status 404 returned error can't find the container with id 947b2cb6f37d167f1ccd6547760e170d6497cc9bc270856bc76ae8ef9f10c488 Nov 01 06:05:21.673885 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:21.673860 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 06:05:21.674906 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:21.674886 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 06:05:21.802540 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:21.802505 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" event={"ID":"adbac784-4ded-4ca3-b985-3e59983fc637","Type":"ContainerStarted","Data":"9a07bd563dd1c68552092f7bc868972a478ddabaeaf86c5d1c83bf7e4f262195"} Nov 01 06:05:21.802998 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:21.802549 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" event={"ID":"adbac784-4ded-4ca3-b985-3e59983fc637","Type":"ContainerStarted","Data":"a94537997b8356bb44c30a3838da4e7732486ca106d4d7fa3b05fe3d95e8a70e"} Nov 01 06:05:21.803994 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:21.803965 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637fa5fc-5g4z9" event={"ID":"3e1dd328-b0f8-4ea0-9aab-93837559f559","Type":"ContainerStarted","Data":"947b2cb6f37d167f1ccd6547760e170d6497cc9bc270856bc76ae8ef9f10c488"} Nov 01 06:05:21.920864 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:21.920804 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" podStartSLOduration=1.920790373 podStartE2EDuration="1.920790373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:05:21.920182908 +0000 UTC m=+1200.575300103" watchObservedRunningTime="2026-11-01 06:05:21.920790373 +0000 UTC m=+1200.575907587" Nov 01 06:05:22.025204 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:22.025162 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" Nov 01 06:05:22.812223 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:22.812190 5040 generic.go:356] "Generic (PLEG): container finished" podID="3e1dd328-b0f8-4ea0-9aab-93837559f559" containerID="475a7c868936f26f22eac7c682b882a932afbd667840098aff1042c7ebe3c368" exitCode=0 Nov 01 06:05:22.812666 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:22.812259 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637fa5fc-5g4z9" event={"ID":"3e1dd328-b0f8-4ea0-9aab-93837559f559","Type":"ContainerDied","Data":"475a7c868936f26f22eac7c682b882a932afbd667840098aff1042c7ebe3c368"} Nov 01 06:05:23.977133 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:23.977106 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637fa5fc-5g4z9" Nov 01 06:05:24.071519 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:24.071418 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e1dd328-b0f8-4ea0-9aab-93837559f559-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e1dd328-b0f8-4ea0-9aab-93837559f559-maas-api-xks-ca\") pod \"3e1dd328-b0f8-4ea0-9aab-93837559f559\" (UID: \"3e1dd328-b0f8-4ea0-9aab-93837559f559\") " Nov 01 06:05:24.071669 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:24.071536 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e1dd328-b0f8-4ea0-9aab-93837559f559-kube-api-access-wbxn5\" (UniqueName: \"kubernetes.io/projected/3e1dd328-b0f8-4ea0-9aab-93837559f559-kube-api-access-wbxn5\") pod \"3e1dd328-b0f8-4ea0-9aab-93837559f559\" (UID: \"3e1dd328-b0f8-4ea0-9aab-93837559f559\") " Nov 01 06:05:24.071707 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:24.071689 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e1dd328-b0f8-4ea0-9aab-93837559f559-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e1dd328-b0f8-4ea0-9aab-93837559f559-maas-api-openshift-ca\") pod \"3e1dd328-b0f8-4ea0-9aab-93837559f559\" (UID: \"3e1dd328-b0f8-4ea0-9aab-93837559f559\") " Nov 01 06:05:24.072053 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:24.072026 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e1dd328-b0f8-4ea0-9aab-93837559f559-maas-api-openshift-ca" pod "3e1dd328-b0f8-4ea0-9aab-93837559f559" (UID: "3e1dd328-b0f8-4ea0-9aab-93837559f559"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:05:24.073639 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:24.073614 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e1dd328-b0f8-4ea0-9aab-93837559f559-maas-api-xks-ca" pod "3e1dd328-b0f8-4ea0-9aab-93837559f559" (UID: "3e1dd328-b0f8-4ea0-9aab-93837559f559"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:05:24.073761 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:24.073700 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e1dd328-b0f8-4ea0-9aab-93837559f559-kube-api-access-wbxn5" pod "3e1dd328-b0f8-4ea0-9aab-93837559f559" (UID: "3e1dd328-b0f8-4ea0-9aab-93837559f559"). InnerVolumeSpecName "kube-api-access-wbxn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:05:24.172782 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:24.172749 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e1dd328-b0f8-4ea0-9aab-93837559f559-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:24.172782 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:24.172779 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbxn5\" (UniqueName: \"kubernetes.io/projected/3e1dd328-b0f8-4ea0-9aab-93837559f559-kube-api-access-wbxn5\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:24.172983 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:24.172790 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e1dd328-b0f8-4ea0-9aab-93837559f559-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:24.828516 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:24.828466 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637fa5fc-5g4z9" Nov 01 06:05:24.828687 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:24.828510 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637fa5fc-5g4z9" event={"ID":"3e1dd328-b0f8-4ea0-9aab-93837559f559","Type":"ContainerDied","Data":"947b2cb6f37d167f1ccd6547760e170d6497cc9bc270856bc76ae8ef9f10c488"} Nov 01 06:05:24.828687 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:24.828567 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="947b2cb6f37d167f1ccd6547760e170d6497cc9bc270856bc76ae8ef9f10c488" Nov 01 06:05:26.455600 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:26.455557 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7d69f54555-8ltqs"] Nov 01 06:05:26.456335 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:26.456314 5040 cpu_manager.go:414] "containerMap: removing container" podUID="3e1dd328-b0f8-4ea0-9aab-93837559f559" containerName="revoke-keys" Nov 01 06:05:26.456431 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:26.456336 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e1dd328-b0f8-4ea0-9aab-93837559f559" containerName="revoke-keys" Nov 01 06:05:26.456536 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:26.456473 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e1dd328-b0f8-4ea0-9aab-93837559f559" containerName="revoke-keys" Nov 01 06:05:26.460334 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:26.460307 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7d69f54555-8ltqs" Nov 01 06:05:26.517410 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:26.517379 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7d69f54555-8ltqs"] Nov 01 06:05:26.563551 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:26.563520 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-58d523\"" Nov 01 06:05:26.563729 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:26.563664 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-58d523-dockercfg-9gjbv\"" Nov 01 06:05:26.612014 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:26.611978 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kplxx\" (UniqueName: \"kubernetes.io/projected/7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd-kube-api-access-kplxx\") pod \"payload-pre-processing-e2e-ipp-a-58d523-7d69f54555-8ltqs\" (UID: \"7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7d69f54555-8ltqs" Nov 01 06:05:26.612190 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:26.612035 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-58d523-7d69f54555-8ltqs\" (UID: \"7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7d69f54555-8ltqs" Nov 01 06:05:26.713303 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:26.713204 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kplxx\" (UniqueName: \"kubernetes.io/projected/7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd-kube-api-access-kplxx\") pod \"payload-pre-processing-e2e-ipp-a-58d523-7d69f54555-8ltqs\" (UID: \"7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7d69f54555-8ltqs" Nov 01 06:05:26.713303 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:26.713259 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-58d523-7d69f54555-8ltqs\" (UID: \"7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7d69f54555-8ltqs" Nov 01 06:05:26.714074 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:26.714052 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-58d523-7d69f54555-8ltqs\" (UID: \"7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7d69f54555-8ltqs" Nov 01 06:05:26.775318 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:26.775284 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kplxx\" (UniqueName: \"kubernetes.io/projected/7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd-kube-api-access-kplxx\") pod \"payload-pre-processing-e2e-ipp-a-58d523-7d69f54555-8ltqs\" (UID: \"7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7d69f54555-8ltqs" Nov 01 06:05:27.073826 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:27.073787 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7d69f54555-8ltqs" Nov 01 06:05:27.257337 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:27.257303 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-58d523-5887dbc76d-7fs98"] Nov 01 06:05:27.262410 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:27.262389 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-58d523-5887dbc76d-7fs98" Nov 01 06:05:27.320444 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:27.320399 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-58d523-5887dbc76d-7fs98"] Nov 01 06:05:27.421466 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:27.421365 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s86nl\" (UniqueName: \"kubernetes.io/projected/feb6d45c-3a57-4a1e-863c-8c0d7972d453-kube-api-access-s86nl\") pod \"payload-processing-e2e-ipp-a-58d523-5887dbc76d-7fs98\" (UID: \"feb6d45c-3a57-4a1e-863c-8c0d7972d453\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-58d523-5887dbc76d-7fs98" Nov 01 06:05:27.421466 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:27.421432 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feb6d45c-3a57-4a1e-863c-8c0d7972d453-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-58d523-5887dbc76d-7fs98\" (UID: \"feb6d45c-3a57-4a1e-863c-8c0d7972d453\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-58d523-5887dbc76d-7fs98" Nov 01 06:05:27.494127 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:27.494089 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7d69f54555-8ltqs"] Nov 01 06:05:27.494599 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:05:27.494245 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b7efe8a_80d1_4b2c_af1b_ae9cbf99d4fd.slice/crio-0b11fa5dbd3770570495856a17881180b127b8a906030eef84f938e53cbb747b WatchSource:0}: Error finding container 0b11fa5dbd3770570495856a17881180b127b8a906030eef84f938e53cbb747b: Status 404 returned error can't find the container with id 0b11fa5dbd3770570495856a17881180b127b8a906030eef84f938e53cbb747b Nov 01 06:05:27.522580 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:27.522547 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s86nl\" (UniqueName: \"kubernetes.io/projected/feb6d45c-3a57-4a1e-863c-8c0d7972d453-kube-api-access-s86nl\") pod \"payload-processing-e2e-ipp-a-58d523-5887dbc76d-7fs98\" (UID: \"feb6d45c-3a57-4a1e-863c-8c0d7972d453\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-58d523-5887dbc76d-7fs98" Nov 01 06:05:27.522764 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:27.522597 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feb6d45c-3a57-4a1e-863c-8c0d7972d453-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-58d523-5887dbc76d-7fs98\" (UID: \"feb6d45c-3a57-4a1e-863c-8c0d7972d453\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-58d523-5887dbc76d-7fs98" Nov 01 06:05:27.523375 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:27.523353 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feb6d45c-3a57-4a1e-863c-8c0d7972d453-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-58d523-5887dbc76d-7fs98\" (UID: \"feb6d45c-3a57-4a1e-863c-8c0d7972d453\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-58d523-5887dbc76d-7fs98" Nov 01 06:05:27.584401 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:27.584366 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s86nl\" (UniqueName: \"kubernetes.io/projected/feb6d45c-3a57-4a1e-863c-8c0d7972d453-kube-api-access-s86nl\") pod \"payload-processing-e2e-ipp-a-58d523-5887dbc76d-7fs98\" (UID: \"feb6d45c-3a57-4a1e-863c-8c0d7972d453\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-58d523-5887dbc76d-7fs98" Nov 01 06:05:27.853915 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:27.853874 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7d69f54555-8ltqs" event={"ID":"7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd","Type":"ContainerStarted","Data":"0b11fa5dbd3770570495856a17881180b127b8a906030eef84f938e53cbb747b"} Nov 01 06:05:27.877217 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:27.877185 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-58d523-5887dbc76d-7fs98" Nov 01 06:05:28.256704 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:05:28.256664 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeb6d45c_3a57_4a1e_863c_8c0d7972d453.slice/crio-ea81b57ca6199e3fc9c7ea0990a05e1bc693a5f0ec35cff97158345556fa79b1 WatchSource:0}: Error finding container ea81b57ca6199e3fc9c7ea0990a05e1bc693a5f0ec35cff97158345556fa79b1: Status 404 returned error can't find the container with id ea81b57ca6199e3fc9c7ea0990a05e1bc693a5f0ec35cff97158345556fa79b1 Nov 01 06:05:28.256857 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:28.256710 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-58d523-5887dbc76d-7fs98"] Nov 01 06:05:28.863291 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:28.863255 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-58d523-5887dbc76d-7fs98" event={"ID":"feb6d45c-3a57-4a1e-863c-8c0d7972d453","Type":"ContainerStarted","Data":"ea81b57ca6199e3fc9c7ea0990a05e1bc693a5f0ec35cff97158345556fa79b1"} Nov 01 06:05:31.030463 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:31.029544 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" Nov 01 06:05:31.262795 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:31.262724 5040 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" podUID="75afe340-0b9c-4b4c-89ab-20d761e6b4f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": context deadline exceeded" Nov 01 06:05:31.385940 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:31.385830 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" podUID="75afe340-0b9c-4b4c-89ab-20d761e6b4f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:05:31.387026 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:31.386178 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" podUID="57cf5e07-36de-48cf-b938-138484cea3a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:05:31.878513 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:31.877873 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" Nov 01 06:05:32.971839 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:32.966165 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" podUID="adbac784-4ded-4ca3-b985-3e59983fc637" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 01 06:05:33.061625 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:33.058895 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" podUID="21f54836-d1ed-4f87-8f26-7c2d939b2590" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:05:33.061625 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:33.060344 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7d69f54555-8ltqs" event={"ID":"7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd","Type":"ContainerStarted","Data":"a95a49ad3c960df2691928169bdd2714803ace0f70dbd14528313282d6c0b350"} Nov 01 06:05:33.061625 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:33.061348 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7d69f54555-8ltqs" Nov 01 06:05:33.194504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:33.193347 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7d69f54555-8ltqs" podStartSLOduration=2.228930719 podStartE2EDuration="7.193324962s" totalImagesPullingTime="4.964394243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:05:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:05:33.185768821 +0000 UTC m=+1211.840885988" watchObservedRunningTime="2026-11-01 06:05:33.193324962 +0000 UTC m=+1211.848442132" Nov 01 06:05:33.358372 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:33.357929 5040 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 06:05:33.358372 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:33.358004 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" podUID="ac11b327-8061-4422-af8d-a37b205d74cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 06:05:33.384984 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:33.384925 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" podUID="75afe340-0b9c-4b4c-89ab-20d761e6b4f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:05:33.388422 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:33.388385 5040 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 06:05:33.389024 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:33.388988 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" podUID="d6aab7db-e2e7-40ef-bf44-6873237c494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 06:05:33.670563 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:33.666941 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" Nov 01 06:05:35.085318 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:35.085277 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-58d523-5887dbc76d-7fs98" event={"ID":"feb6d45c-3a57-4a1e-863c-8c0d7972d453","Type":"ContainerStarted","Data":"20407a10f1597e26ec22f3ab05ab31043b436bca555362dc37e5133883ccd838"} Nov 01 06:05:35.085745 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:35.085381 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-58d523-5887dbc76d-7fs98" Nov 01 06:05:35.205637 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:35.205581 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-58d523-5887dbc76d-7fs98" podStartSLOduration=2.467933632 podStartE2EDuration="8.205566239s" totalImagesPullingTime="5.737632607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:05:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:05:35.202466108 +0000 UTC m=+1213.857583279" watchObservedRunningTime="2026-11-01 06:05:35.205566239 +0000 UTC m=+1213.860683406" Nov 01 06:05:39.976438 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:05:39.976397 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:05:45.087848 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:45.087815 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7d69f54555-8ltqs" Nov 01 06:05:46.094226 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:46.094195 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-58d523-5887dbc76d-7fs98" Nov 01 06:05:47.655804 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:47.655765 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e729141-x95rn"] Nov 01 06:05:47.659940 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:47.659920 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e729141-x95rn" Nov 01 06:05:47.717770 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:47.717735 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e729141-x95rn"] Nov 01 06:05:47.736162 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:47.736121 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cb7101b-6199-46dc-a4b0-a7832ec19011-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e729141-x95rn\" (UID: \"2cb7101b-6199-46dc-a4b0-a7832ec19011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e729141-x95rn" Nov 01 06:05:47.736330 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:47.736282 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p4lp\" (UniqueName: \"kubernetes.io/projected/2cb7101b-6199-46dc-a4b0-a7832ec19011-kube-api-access-2p4lp\") pod \"maas-api-revoke-sub-0e729141-x95rn\" (UID: \"2cb7101b-6199-46dc-a4b0-a7832ec19011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e729141-x95rn" Nov 01 06:05:47.736330 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:47.736317 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cb7101b-6199-46dc-a4b0-a7832ec19011-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e729141-x95rn\" (UID: \"2cb7101b-6199-46dc-a4b0-a7832ec19011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e729141-x95rn" Nov 01 06:05:47.763696 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:47.763662 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:05:47.763871 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:47.763723 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:05:47.763871 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:47.763794 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:05:47.763871 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:47.763822 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:05:47.838049 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:47.838012 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cb7101b-6199-46dc-a4b0-a7832ec19011-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e729141-x95rn\" (UID: \"2cb7101b-6199-46dc-a4b0-a7832ec19011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e729141-x95rn" Nov 01 06:05:47.838399 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:47.838368 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p4lp\" (UniqueName: \"kubernetes.io/projected/2cb7101b-6199-46dc-a4b0-a7832ec19011-kube-api-access-2p4lp\") pod \"maas-api-revoke-sub-0e729141-x95rn\" (UID: \"2cb7101b-6199-46dc-a4b0-a7832ec19011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e729141-x95rn" Nov 01 06:05:47.838545 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:47.838418 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cb7101b-6199-46dc-a4b0-a7832ec19011-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e729141-x95rn\" (UID: \"2cb7101b-6199-46dc-a4b0-a7832ec19011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e729141-x95rn" Nov 01 06:05:47.839259 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:47.839237 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cb7101b-6199-46dc-a4b0-a7832ec19011-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e729141-x95rn\" (UID: \"2cb7101b-6199-46dc-a4b0-a7832ec19011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e729141-x95rn" Nov 01 06:05:47.841420 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:47.841397 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cb7101b-6199-46dc-a4b0-a7832ec19011-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e729141-x95rn\" (UID: \"2cb7101b-6199-46dc-a4b0-a7832ec19011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e729141-x95rn" Nov 01 06:05:47.900584 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:47.900546 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p4lp\" (UniqueName: \"kubernetes.io/projected/2cb7101b-6199-46dc-a4b0-a7832ec19011-kube-api-access-2p4lp\") pod \"maas-api-revoke-sub-0e729141-x95rn\" (UID: \"2cb7101b-6199-46dc-a4b0-a7832ec19011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e729141-x95rn" Nov 01 06:05:47.973346 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:47.973264 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e729141-x95rn" Nov 01 06:05:48.351911 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:48.351873 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e729141-x95rn"] Nov 01 06:05:48.352112 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:05:48.352078 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb7101b_6199_46dc_a4b0_a7832ec19011.slice/crio-a9be047d5b936df09de25d7d09048f86ceb823f31a5be316aaf12f11f3c8ca59 WatchSource:0}: Error finding container a9be047d5b936df09de25d7d09048f86ceb823f31a5be316aaf12f11f3c8ca59: Status 404 returned error can't find the container with id a9be047d5b936df09de25d7d09048f86ceb823f31a5be316aaf12f11f3c8ca59 Nov 01 06:05:49.187777 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:49.187740 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e729141-x95rn" event={"ID":"2cb7101b-6199-46dc-a4b0-a7832ec19011","Type":"ContainerStarted","Data":"1c380dfa9c6d6ee8dae542b8325be07e3d8fcb13ed154004b5cb4d45adb63125"} Nov 01 06:05:49.188252 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:49.187795 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e729141-x95rn" event={"ID":"2cb7101b-6199-46dc-a4b0-a7832ec19011","Type":"ContainerStarted","Data":"a9be047d5b936df09de25d7d09048f86ceb823f31a5be316aaf12f11f3c8ca59"} Nov 01 06:05:49.307282 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:49.307228 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e729141-x95rn" podStartSLOduration=2.307213565 podStartE2EDuration="2.307213565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:05:49.304251003 +0000 UTC m=+1227.959368171" watchObservedRunningTime="2026-11-01 06:05:49.307213565 +0000 UTC m=+1227.962330731" Nov 01 06:05:50.196119 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:50.196083 5040 generic.go:356] "Generic (PLEG): container finished" podID="2cb7101b-6199-46dc-a4b0-a7832ec19011" containerID="1c380dfa9c6d6ee8dae542b8325be07e3d8fcb13ed154004b5cb4d45adb63125" exitCode=0 Nov 01 06:05:50.196664 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:50.196164 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e729141-x95rn" event={"ID":"2cb7101b-6199-46dc-a4b0-a7832ec19011","Type":"ContainerDied","Data":"1c380dfa9c6d6ee8dae542b8325be07e3d8fcb13ed154004b5cb4d45adb63125"} Nov 01 06:05:51.491385 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:51.491322 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e729141-x95rn" Nov 01 06:05:51.636865 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:51.636570 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cb7101b-6199-46dc-a4b0-a7832ec19011-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cb7101b-6199-46dc-a4b0-a7832ec19011-maas-api-openshift-ca\") pod \"2cb7101b-6199-46dc-a4b0-a7832ec19011\" (UID: \"2cb7101b-6199-46dc-a4b0-a7832ec19011\") " Nov 01 06:05:51.636865 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:51.636689 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cb7101b-6199-46dc-a4b0-a7832ec19011-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cb7101b-6199-46dc-a4b0-a7832ec19011-maas-api-xks-ca\") pod \"2cb7101b-6199-46dc-a4b0-a7832ec19011\" (UID: \"2cb7101b-6199-46dc-a4b0-a7832ec19011\") " Nov 01 06:05:51.636865 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:51.636859 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cb7101b-6199-46dc-a4b0-a7832ec19011-kube-api-access-2p4lp\" (UniqueName: \"kubernetes.io/projected/2cb7101b-6199-46dc-a4b0-a7832ec19011-kube-api-access-2p4lp\") pod \"2cb7101b-6199-46dc-a4b0-a7832ec19011\" (UID: \"2cb7101b-6199-46dc-a4b0-a7832ec19011\") " Nov 01 06:05:51.637982 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:51.637948 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cb7101b-6199-46dc-a4b0-a7832ec19011-maas-api-openshift-ca" pod "2cb7101b-6199-46dc-a4b0-a7832ec19011" (UID: "2cb7101b-6199-46dc-a4b0-a7832ec19011"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:05:51.644646 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:51.644596 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb7101b-6199-46dc-a4b0-a7832ec19011-kube-api-access-2p4lp" pod "2cb7101b-6199-46dc-a4b0-a7832ec19011" (UID: "2cb7101b-6199-46dc-a4b0-a7832ec19011"). InnerVolumeSpecName "kube-api-access-2p4lp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:05:51.645601 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:51.645562 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cb7101b-6199-46dc-a4b0-a7832ec19011-maas-api-xks-ca" pod "2cb7101b-6199-46dc-a4b0-a7832ec19011" (UID: "2cb7101b-6199-46dc-a4b0-a7832ec19011"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:05:51.738471 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:51.738361 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p4lp\" (UniqueName: \"kubernetes.io/projected/2cb7101b-6199-46dc-a4b0-a7832ec19011-kube-api-access-2p4lp\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:51.738471 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:51.738412 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cb7101b-6199-46dc-a4b0-a7832ec19011-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:51.738471 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:51.738429 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cb7101b-6199-46dc-a4b0-a7832ec19011-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:52.234520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:52.234458 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e729141-x95rn" event={"ID":"2cb7101b-6199-46dc-a4b0-a7832ec19011","Type":"ContainerDied","Data":"a9be047d5b936df09de25d7d09048f86ceb823f31a5be316aaf12f11f3c8ca59"} Nov 01 06:05:52.234710 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:52.234535 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e729141-x95rn" Nov 01 06:05:52.234710 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:52.234553 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9be047d5b936df09de25d7d09048f86ceb823f31a5be316aaf12f11f3c8ca59" Nov 01 06:05:58.445545 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:58.445494 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cb9816e-b97hn"] Nov 01 06:05:58.446096 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:58.446081 5040 cpu_manager.go:414] "containerMap: removing container" podUID="2cb7101b-6199-46dc-a4b0-a7832ec19011" containerName="revoke-keys" Nov 01 06:05:58.446148 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:58.446098 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cb7101b-6199-46dc-a4b0-a7832ec19011" containerName="revoke-keys" Nov 01 06:05:58.446225 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:58.446208 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cb7101b-6199-46dc-a4b0-a7832ec19011" containerName="revoke-keys" Nov 01 06:05:58.450972 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:58.450952 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb9816e-b97hn" Nov 01 06:05:58.507792 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:58.507763 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cb9816e-b97hn"] Nov 01 06:05:58.543100 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:58.543053 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/629b293f-5a09-48c0-b154-85d8c81ca510-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cb9816e-b97hn\" (UID: \"629b293f-5a09-48c0-b154-85d8c81ca510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb9816e-b97hn" Nov 01 06:05:58.543100 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:58.543102 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grqp2\" (UniqueName: \"kubernetes.io/projected/629b293f-5a09-48c0-b154-85d8c81ca510-kube-api-access-grqp2\") pod \"maas-api-revoke-sub-0cb9816e-b97hn\" (UID: \"629b293f-5a09-48c0-b154-85d8c81ca510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb9816e-b97hn" Nov 01 06:05:58.543425 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:58.543251 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/629b293f-5a09-48c0-b154-85d8c81ca510-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cb9816e-b97hn\" (UID: \"629b293f-5a09-48c0-b154-85d8c81ca510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb9816e-b97hn" Nov 01 06:05:58.553058 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:58.553008 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:05:58.553058 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:58.553012 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:05:58.553058 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:58.553051 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:05:58.554464 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:58.554440 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:05:58.644782 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:58.644743 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/629b293f-5a09-48c0-b154-85d8c81ca510-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cb9816e-b97hn\" (UID: \"629b293f-5a09-48c0-b154-85d8c81ca510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb9816e-b97hn" Nov 01 06:05:58.644982 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:58.644792 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grqp2\" (UniqueName: \"kubernetes.io/projected/629b293f-5a09-48c0-b154-85d8c81ca510-kube-api-access-grqp2\") pod \"maas-api-revoke-sub-0cb9816e-b97hn\" (UID: \"629b293f-5a09-48c0-b154-85d8c81ca510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb9816e-b97hn" Nov 01 06:05:58.644982 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:58.644821 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/629b293f-5a09-48c0-b154-85d8c81ca510-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cb9816e-b97hn\" (UID: \"629b293f-5a09-48c0-b154-85d8c81ca510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb9816e-b97hn" Nov 01 06:05:58.645593 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:58.645568 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/629b293f-5a09-48c0-b154-85d8c81ca510-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cb9816e-b97hn\" (UID: \"629b293f-5a09-48c0-b154-85d8c81ca510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb9816e-b97hn" Nov 01 06:05:58.647162 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:58.647145 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/629b293f-5a09-48c0-b154-85d8c81ca510-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cb9816e-b97hn\" (UID: \"629b293f-5a09-48c0-b154-85d8c81ca510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb9816e-b97hn" Nov 01 06:05:58.706925 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:58.706850 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grqp2\" (UniqueName: \"kubernetes.io/projected/629b293f-5a09-48c0-b154-85d8c81ca510-kube-api-access-grqp2\") pod \"maas-api-revoke-sub-0cb9816e-b97hn\" (UID: \"629b293f-5a09-48c0-b154-85d8c81ca510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb9816e-b97hn" Nov 01 06:05:58.767247 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:58.767208 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb9816e-b97hn" Nov 01 06:05:59.006335 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:59.006299 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c18e3e98-j984d"] Nov 01 06:05:59.011840 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:59.011818 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c18e3e98-j984d" Nov 01 06:05:59.068905 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:59.068871 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c18e3e98-j984d"] Nov 01 06:05:59.151234 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:59.151193 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8892735-cb03-43a6-8312-808304b332a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c18e3e98-j984d\" (UID: \"f8892735-cb03-43a6-8312-808304b332a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c18e3e98-j984d" Nov 01 06:05:59.151414 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:59.151338 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww272\" (UniqueName: \"kubernetes.io/projected/f8892735-cb03-43a6-8312-808304b332a3-kube-api-access-ww272\") pod \"maas-api-revoke-sub-c18e3e98-j984d\" (UID: \"f8892735-cb03-43a6-8312-808304b332a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c18e3e98-j984d" Nov 01 06:05:59.151414 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:59.151396 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8892735-cb03-43a6-8312-808304b332a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c18e3e98-j984d\" (UID: \"f8892735-cb03-43a6-8312-808304b332a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c18e3e98-j984d" Nov 01 06:05:59.179257 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:59.179223 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cb9816e-b97hn"] Nov 01 06:05:59.179558 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:05:59.179524 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod629b293f_5a09_48c0_b154_85d8c81ca510.slice/crio-83592c4177f2327315818cf0528b9149b4123b5bb9ccafeb52fb46957ea2339c WatchSource:0}: Error finding container 83592c4177f2327315818cf0528b9149b4123b5bb9ccafeb52fb46957ea2339c: Status 404 returned error can't find the container with id 83592c4177f2327315818cf0528b9149b4123b5bb9ccafeb52fb46957ea2339c Nov 01 06:05:59.252722 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:59.252682 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8892735-cb03-43a6-8312-808304b332a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c18e3e98-j984d\" (UID: \"f8892735-cb03-43a6-8312-808304b332a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c18e3e98-j984d" Nov 01 06:05:59.252932 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:59.252852 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww272\" (UniqueName: \"kubernetes.io/projected/f8892735-cb03-43a6-8312-808304b332a3-kube-api-access-ww272\") pod \"maas-api-revoke-sub-c18e3e98-j984d\" (UID: \"f8892735-cb03-43a6-8312-808304b332a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c18e3e98-j984d" Nov 01 06:05:59.252932 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:59.252905 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8892735-cb03-43a6-8312-808304b332a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c18e3e98-j984d\" (UID: \"f8892735-cb03-43a6-8312-808304b332a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c18e3e98-j984d" Nov 01 06:05:59.253416 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:59.253394 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8892735-cb03-43a6-8312-808304b332a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c18e3e98-j984d\" (UID: \"f8892735-cb03-43a6-8312-808304b332a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c18e3e98-j984d" Nov 01 06:05:59.255288 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:59.255270 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8892735-cb03-43a6-8312-808304b332a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c18e3e98-j984d\" (UID: \"f8892735-cb03-43a6-8312-808304b332a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c18e3e98-j984d" Nov 01 06:05:59.311073 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:59.310986 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb9816e-b97hn" event={"ID":"629b293f-5a09-48c0-b154-85d8c81ca510","Type":"ContainerStarted","Data":"83592c4177f2327315818cf0528b9149b4123b5bb9ccafeb52fb46957ea2339c"} Nov 01 06:05:59.314781 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:59.314750 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww272\" (UniqueName: \"kubernetes.io/projected/f8892735-cb03-43a6-8312-808304b332a3-kube-api-access-ww272\") pod \"maas-api-revoke-sub-c18e3e98-j984d\" (UID: \"f8892735-cb03-43a6-8312-808304b332a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c18e3e98-j984d" Nov 01 06:05:59.326559 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:59.326528 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c18e3e98-j984d" Nov 01 06:05:59.708677 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:05:59.708446 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c18e3e98-j984d"] Nov 01 06:05:59.709130 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:05:59.708674 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8892735_cb03_43a6_8312_808304b332a3.slice/crio-7c2fc249152c57625cb687cf8485b88359c7275e6baa56aebbc41254dcc5f673 WatchSource:0}: Error finding container 7c2fc249152c57625cb687cf8485b88359c7275e6baa56aebbc41254dcc5f673: Status 404 returned error can't find the container with id 7c2fc249152c57625cb687cf8485b88359c7275e6baa56aebbc41254dcc5f673 Nov 01 06:06:00.322404 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.322359 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c18e3e98-j984d" event={"ID":"f8892735-cb03-43a6-8312-808304b332a3","Type":"ContainerStarted","Data":"83bf0052bd200b3c42c86db55737b383e70c43e85dc4d47744304afd58fb9a3c"} Nov 01 06:06:00.322705 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.322465 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c18e3e98-j984d" event={"ID":"f8892735-cb03-43a6-8312-808304b332a3","Type":"ContainerStarted","Data":"7c2fc249152c57625cb687cf8485b88359c7275e6baa56aebbc41254dcc5f673"} Nov 01 06:06:00.324516 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.324466 5040 generic.go:356] "Generic (PLEG): container finished" podID="629b293f-5a09-48c0-b154-85d8c81ca510" containerID="acdf6aa47dd3b4eff1b841a8dacef5d3910ba952af6f251b786fff8bd62d4c92" exitCode=0 Nov 01 06:06:00.324650 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.324525 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb9816e-b97hn" event={"ID":"629b293f-5a09-48c0-b154-85d8c81ca510","Type":"ContainerDied","Data":"acdf6aa47dd3b4eff1b841a8dacef5d3910ba952af6f251b786fff8bd62d4c92"} Nov 01 06:06:00.326394 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.326364 5040 generic.go:356] "Generic (PLEG): container finished" podID="0073efde-c913-424b-93e8-bda25a24a209" containerID="73a7065d2e4f82ade11bd5eba20661c60ba5233484bfa20430f66a700499a859" exitCode=0 Nov 01 06:06:00.326534 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.326431 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" event={"ID":"0073efde-c913-424b-93e8-bda25a24a209","Type":"ContainerDied","Data":"73a7065d2e4f82ade11bd5eba20661c60ba5233484bfa20430f66a700499a859"} Nov 01 06:06:00.326853 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.326836 5040 scope.go:122] "RemoveContainer" containerID="73a7065d2e4f82ade11bd5eba20661c60ba5233484bfa20430f66a700499a859" Nov 01 06:06:00.328449 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.328422 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:06:00.328533 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.328509 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:06:00.328581 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.328543 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:06:00.442627 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.442568 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c18e3e98-j984d" podStartSLOduration=2.442554642 podStartE2EDuration="2.442554642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:05:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:06:00.438145219 +0000 UTC m=+1239.093262389" watchObservedRunningTime="2026-11-01 06:06:00.442554642 +0000 UTC m=+1239.097671808" Nov 01 06:06:00.554992 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.554952 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26"] Nov 01 06:06:00.558692 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.558673 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" Nov 01 06:06:00.630453 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.627922 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26"] Nov 01 06:06:00.671165 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.671114 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1442af5-c9a0-4995-be20-1f65aa32d0bd-istio-data\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26\" (UID: \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" Nov 01 06:06:00.671359 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.671184 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1442af5-c9a0-4995-be20-1f65aa32d0bd-istiod-ca-cert\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26\" (UID: \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" Nov 01 06:06:00.671359 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.671223 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpdx7\" (UniqueName: \"kubernetes.io/projected/e1442af5-c9a0-4995-be20-1f65aa32d0bd-kube-api-access-lpdx7\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26\" (UID: \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" Nov 01 06:06:00.671359 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.671274 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1442af5-c9a0-4995-be20-1f65aa32d0bd-istio-envoy\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26\" (UID: \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" Nov 01 06:06:00.671359 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.671313 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1442af5-c9a0-4995-be20-1f65aa32d0bd-istio-podinfo\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26\" (UID: \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" Nov 01 06:06:00.671359 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.671341 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1442af5-c9a0-4995-be20-1f65aa32d0bd-workload-socket\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26\" (UID: \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" Nov 01 06:06:00.671658 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.671403 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1442af5-c9a0-4995-be20-1f65aa32d0bd-workload-certs\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26\" (UID: \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" Nov 01 06:06:00.671658 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.671425 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1442af5-c9a0-4995-be20-1f65aa32d0bd-istio-token\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26\" (UID: \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" Nov 01 06:06:00.671658 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.671524 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1442af5-c9a0-4995-be20-1f65aa32d0bd-credential-socket\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26\" (UID: \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" Nov 01 06:06:00.772512 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.772436 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1442af5-c9a0-4995-be20-1f65aa32d0bd-istio-data\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26\" (UID: \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" Nov 01 06:06:00.772961 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.772528 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1442af5-c9a0-4995-be20-1f65aa32d0bd-istiod-ca-cert\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26\" (UID: \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" Nov 01 06:06:00.772961 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.772567 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpdx7\" (UniqueName: \"kubernetes.io/projected/e1442af5-c9a0-4995-be20-1f65aa32d0bd-kube-api-access-lpdx7\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26\" (UID: \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" Nov 01 06:06:00.772961 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.772618 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1442af5-c9a0-4995-be20-1f65aa32d0bd-istio-envoy\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26\" (UID: \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" Nov 01 06:06:00.772961 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.772653 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1442af5-c9a0-4995-be20-1f65aa32d0bd-istio-podinfo\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26\" (UID: \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" Nov 01 06:06:00.772961 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.772678 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1442af5-c9a0-4995-be20-1f65aa32d0bd-workload-socket\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26\" (UID: \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" Nov 01 06:06:00.772961 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.772738 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1442af5-c9a0-4995-be20-1f65aa32d0bd-workload-certs\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26\" (UID: \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" Nov 01 06:06:00.772961 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.772763 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1442af5-c9a0-4995-be20-1f65aa32d0bd-istio-token\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26\" (UID: \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" Nov 01 06:06:00.772961 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.772840 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1442af5-c9a0-4995-be20-1f65aa32d0bd-credential-socket\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26\" (UID: \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" Nov 01 06:06:00.772961 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.772890 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1442af5-c9a0-4995-be20-1f65aa32d0bd-istio-data\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26\" (UID: \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" Nov 01 06:06:00.773380 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.773188 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1442af5-c9a0-4995-be20-1f65aa32d0bd-credential-socket\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26\" (UID: \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" Nov 01 06:06:00.773380 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.773239 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1442af5-c9a0-4995-be20-1f65aa32d0bd-workload-socket\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26\" (UID: \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" Nov 01 06:06:00.773449 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.773382 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1442af5-c9a0-4995-be20-1f65aa32d0bd-istiod-ca-cert\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26\" (UID: \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" Nov 01 06:06:00.773527 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.773463 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1442af5-c9a0-4995-be20-1f65aa32d0bd-workload-certs\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26\" (UID: \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" Nov 01 06:06:00.775273 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.775246 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1442af5-c9a0-4995-be20-1f65aa32d0bd-istio-envoy\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26\" (UID: \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" Nov 01 06:06:00.775634 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.775613 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1442af5-c9a0-4995-be20-1f65aa32d0bd-istio-podinfo\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26\" (UID: \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" Nov 01 06:06:00.834324 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.834286 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1442af5-c9a0-4995-be20-1f65aa32d0bd-istio-token\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26\" (UID: \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" Nov 01 06:06:00.834887 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.834859 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpdx7\" (UniqueName: \"kubernetes.io/projected/e1442af5-c9a0-4995-be20-1f65aa32d0bd-kube-api-access-lpdx7\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26\" (UID: \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" Nov 01 06:06:00.929084 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:00.928996 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" Nov 01 06:06:01.334573 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:01.334539 5040 generic.go:356] "Generic (PLEG): container finished" podID="f8892735-cb03-43a6-8312-808304b332a3" containerID="83bf0052bd200b3c42c86db55737b383e70c43e85dc4d47744304afd58fb9a3c" exitCode=0 Nov 01 06:06:01.334772 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:01.334631 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c18e3e98-j984d" event={"ID":"f8892735-cb03-43a6-8312-808304b332a3","Type":"ContainerDied","Data":"83bf0052bd200b3c42c86db55737b383e70c43e85dc4d47744304afd58fb9a3c"} Nov 01 06:06:01.336752 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:01.336726 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" event={"ID":"0073efde-c913-424b-93e8-bda25a24a209","Type":"ContainerStarted","Data":"a43b59ddb22ed0bec2898be848b183d8bbb19b5ebf26279a1087e91bbd9b435e"} Nov 01 06:06:01.382174 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:01.382140 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26"] Nov 01 06:06:01.382895 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:06:01.382874 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1442af5_c9a0_4995_be20_1f65aa32d0bd.slice/crio-6e895b55a77bcb284f5f21015996cdf80e082010c583d700bf89931967f38564 WatchSource:0}: Error finding container 6e895b55a77bcb284f5f21015996cdf80e082010c583d700bf89931967f38564: Status 404 returned error can't find the container with id 6e895b55a77bcb284f5f21015996cdf80e082010c583d700bf89931967f38564 Nov 01 06:06:01.386160 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:01.386104 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:06:01.386239 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:01.386211 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:06:01.386274 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:01.386251 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:06:01.536190 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:01.536162 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb9816e-b97hn" Nov 01 06:06:01.583690 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:01.583650 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/629b293f-5a09-48c0-b154-85d8c81ca510-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/629b293f-5a09-48c0-b154-85d8c81ca510-maas-api-xks-ca\") pod \"629b293f-5a09-48c0-b154-85d8c81ca510\" (UID: \"629b293f-5a09-48c0-b154-85d8c81ca510\") " Nov 01 06:06:01.583901 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:01.583825 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/629b293f-5a09-48c0-b154-85d8c81ca510-kube-api-access-grqp2\" (UniqueName: \"kubernetes.io/projected/629b293f-5a09-48c0-b154-85d8c81ca510-kube-api-access-grqp2\") pod \"629b293f-5a09-48c0-b154-85d8c81ca510\" (UID: \"629b293f-5a09-48c0-b154-85d8c81ca510\") " Nov 01 06:06:01.583901 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:01.583876 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/629b293f-5a09-48c0-b154-85d8c81ca510-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/629b293f-5a09-48c0-b154-85d8c81ca510-maas-api-openshift-ca\") pod \"629b293f-5a09-48c0-b154-85d8c81ca510\" (UID: \"629b293f-5a09-48c0-b154-85d8c81ca510\") " Nov 01 06:06:01.584438 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:01.584392 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/629b293f-5a09-48c0-b154-85d8c81ca510-maas-api-openshift-ca" pod "629b293f-5a09-48c0-b154-85d8c81ca510" (UID: "629b293f-5a09-48c0-b154-85d8c81ca510"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:06:01.586518 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:01.586448 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629b293f-5a09-48c0-b154-85d8c81ca510-kube-api-access-grqp2" pod "629b293f-5a09-48c0-b154-85d8c81ca510" (UID: "629b293f-5a09-48c0-b154-85d8c81ca510"). InnerVolumeSpecName "kube-api-access-grqp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:06:01.587862 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:01.587840 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/629b293f-5a09-48c0-b154-85d8c81ca510-maas-api-xks-ca" pod "629b293f-5a09-48c0-b154-85d8c81ca510" (UID: "629b293f-5a09-48c0-b154-85d8c81ca510"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:06:01.685656 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:01.685622 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grqp2\" (UniqueName: \"kubernetes.io/projected/629b293f-5a09-48c0-b154-85d8c81ca510-kube-api-access-grqp2\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:01.685656 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:01.685653 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/629b293f-5a09-48c0-b154-85d8c81ca510-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:01.685864 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:01.685667 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/629b293f-5a09-48c0-b154-85d8c81ca510-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:02.167229 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:02.167193 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:06:02.168120 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:02.168091 5040 patch_prober.go:29] interesting pod/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 01 06:06:02.168231 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:02.168144 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" podUID="0073efde-c913-424b-93e8-bda25a24a209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 01 06:06:02.369515 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:02.369467 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb9816e-b97hn" Nov 01 06:06:02.369515 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:02.369471 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb9816e-b97hn" event={"ID":"629b293f-5a09-48c0-b154-85d8c81ca510","Type":"ContainerDied","Data":"83592c4177f2327315818cf0528b9149b4123b5bb9ccafeb52fb46957ea2339c"} Nov 01 06:06:02.369733 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:02.369553 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83592c4177f2327315818cf0528b9149b4123b5bb9ccafeb52fb46957ea2339c" Nov 01 06:06:02.374535 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:02.372416 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" event={"ID":"e1442af5-c9a0-4995-be20-1f65aa32d0bd","Type":"ContainerStarted","Data":"782f05aa12faf7e0e9102a5aeab379905c6d5d8d7095cd270d679eb3a4019a72"} Nov 01 06:06:02.374535 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:02.372497 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" event={"ID":"e1442af5-c9a0-4995-be20-1f65aa32d0bd","Type":"ContainerStarted","Data":"6e895b55a77bcb284f5f21015996cdf80e082010c583d700bf89931967f38564"} Nov 01 06:06:02.449575 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:02.449462 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:06:02.556318 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:02.555192 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" podStartSLOduration=2.5551737709999998 podStartE2EDuration="2.555173771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:06:02.501143519 +0000 UTC m=+1241.156260687" watchObservedRunningTime="2026-11-01 06:06:02.555173771 +0000 UTC m=+1241.210290935" Nov 01 06:06:02.657025 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:02.657000 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c18e3e98-j984d" Nov 01 06:06:02.757099 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:02.757054 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8892735-cb03-43a6-8312-808304b332a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8892735-cb03-43a6-8312-808304b332a3-maas-api-openshift-ca\") pod \"f8892735-cb03-43a6-8312-808304b332a3\" (UID: \"f8892735-cb03-43a6-8312-808304b332a3\") " Nov 01 06:06:02.757318 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:02.757187 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8892735-cb03-43a6-8312-808304b332a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8892735-cb03-43a6-8312-808304b332a3-maas-api-xks-ca\") pod \"f8892735-cb03-43a6-8312-808304b332a3\" (UID: \"f8892735-cb03-43a6-8312-808304b332a3\") " Nov 01 06:06:02.757318 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:02.757266 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8892735-cb03-43a6-8312-808304b332a3-kube-api-access-ww272\" (UniqueName: \"kubernetes.io/projected/f8892735-cb03-43a6-8312-808304b332a3-kube-api-access-ww272\") pod \"f8892735-cb03-43a6-8312-808304b332a3\" (UID: \"f8892735-cb03-43a6-8312-808304b332a3\") " Nov 01 06:06:02.757435 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:02.757403 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8892735-cb03-43a6-8312-808304b332a3-maas-api-openshift-ca" pod "f8892735-cb03-43a6-8312-808304b332a3" (UID: "f8892735-cb03-43a6-8312-808304b332a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:06:02.757725 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:02.757705 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8892735-cb03-43a6-8312-808304b332a3-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:02.759221 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:02.759198 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8892735-cb03-43a6-8312-808304b332a3-maas-api-xks-ca" pod "f8892735-cb03-43a6-8312-808304b332a3" (UID: "f8892735-cb03-43a6-8312-808304b332a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:06:02.759406 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:02.759388 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8892735-cb03-43a6-8312-808304b332a3-kube-api-access-ww272" pod "f8892735-cb03-43a6-8312-808304b332a3" (UID: "f8892735-cb03-43a6-8312-808304b332a3"). InnerVolumeSpecName "kube-api-access-ww272". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:06:02.859265 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:02.859227 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww272\" (UniqueName: \"kubernetes.io/projected/f8892735-cb03-43a6-8312-808304b332a3-kube-api-access-ww272\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:02.859265 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:02.859259 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8892735-cb03-43a6-8312-808304b332a3-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:02.934509 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:02.931143 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" Nov 01 06:06:02.934509 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:02.931277 5040 patch_prober.go:29] interesting pod/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 01 06:06:02.934509 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:02.931323 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" podUID="e1442af5-c9a0-4995-be20-1f65aa32d0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 01 06:06:03.167618 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:03.167540 5040 patch_prober.go:29] interesting pod/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 01 06:06:03.168116 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:03.167777 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" podUID="0073efde-c913-424b-93e8-bda25a24a209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 01 06:06:03.379304 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:03.379273 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c18e3e98-j984d" event={"ID":"f8892735-cb03-43a6-8312-808304b332a3","Type":"ContainerDied","Data":"7c2fc249152c57625cb687cf8485b88359c7275e6baa56aebbc41254dcc5f673"} Nov 01 06:06:03.379528 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:03.379512 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c2fc249152c57625cb687cf8485b88359c7275e6baa56aebbc41254dcc5f673" Nov 01 06:06:03.379599 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:03.379326 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c18e3e98-j984d" Nov 01 06:06:03.935339 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:03.935297 5040 patch_prober.go:29] interesting pod/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 01 06:06:03.936209 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:03.936173 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" podUID="e1442af5-c9a0-4995-be20-1f65aa32d0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 01 06:06:04.166890 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:04.166843 5040 patch_prober.go:29] interesting pod/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 01 06:06:04.167068 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:04.166906 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" podUID="0073efde-c913-424b-93e8-bda25a24a209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 01 06:06:04.936930 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:04.936888 5040 patch_prober.go:29] interesting pod/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 01 06:06:04.937529 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:04.937495 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" podUID="e1442af5-c9a0-4995-be20-1f65aa32d0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 01 06:06:05.167390 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:05.167355 5040 patch_prober.go:29] interesting pod/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 01 06:06:05.167577 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:05.167406 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" podUID="0073efde-c913-424b-93e8-bda25a24a209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 01 06:06:05.940518 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:05.938623 5040 patch_prober.go:29] interesting pod/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 01 06:06:05.940518 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:05.938692 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" podUID="e1442af5-c9a0-4995-be20-1f65aa32d0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 01 06:06:06.167281 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:06.167152 5040 patch_prober.go:29] interesting pod/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 01 06:06:06.167281 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:06.167211 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" podUID="0073efde-c913-424b-93e8-bda25a24a209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 01 06:06:06.929598 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:06.929550 5040 patch_prober.go:29] interesting pod/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 01 06:06:06.929816 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:06.929617 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" podUID="e1442af5-c9a0-4995-be20-1f65aa32d0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 01 06:06:07.172074 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:07.172043 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:06:07.173137 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:07.173120 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:06:07.934293 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:07.934262 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" Nov 01 06:06:07.934666 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:07.934647 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" Nov 01 06:06:07.935366 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:07.935349 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" Nov 01 06:06:10.037755 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:06:10.037716 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:06:12.108097 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:12.108061 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ffc4cba-nd6dz"] Nov 01 06:06:12.110049 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:12.110023 5040 cpu_manager.go:414] "containerMap: removing container" podUID="629b293f-5a09-48c0-b154-85d8c81ca510" containerName="revoke-keys" Nov 01 06:06:12.110213 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:12.110200 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="629b293f-5a09-48c0-b154-85d8c81ca510" containerName="revoke-keys" Nov 01 06:06:12.110328 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:12.110318 5040 cpu_manager.go:414] "containerMap: removing container" podUID="f8892735-cb03-43a6-8312-808304b332a3" containerName="revoke-keys" Nov 01 06:06:12.110400 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:12.110391 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8892735-cb03-43a6-8312-808304b332a3" containerName="revoke-keys" Nov 01 06:06:12.110624 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:12.110613 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8892735-cb03-43a6-8312-808304b332a3" containerName="revoke-keys" Nov 01 06:06:12.110723 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:12.110714 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="629b293f-5a09-48c0-b154-85d8c81ca510" containerName="revoke-keys" Nov 01 06:06:12.115053 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:12.115034 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ffc4cba-nd6dz" Nov 01 06:06:12.235531 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:12.219136 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:06:12.268473 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:12.265645 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:06:12.268473 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:12.265758 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:06:12.268473 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:12.265851 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:06:12.273816 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:12.272982 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ffc4cba-nd6dz"] Nov 01 06:06:12.287818 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:12.287774 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ae36837-6f06-4bfb-87b1-4bcfed69167e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ffc4cba-nd6dz\" (UID: \"7ae36837-6f06-4bfb-87b1-4bcfed69167e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ffc4cba-nd6dz" Nov 01 06:06:12.288443 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:12.288415 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ae36837-6f06-4bfb-87b1-4bcfed69167e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ffc4cba-nd6dz\" (UID: \"7ae36837-6f06-4bfb-87b1-4bcfed69167e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ffc4cba-nd6dz" Nov 01 06:06:12.288995 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:12.288972 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59tbn\" (UniqueName: \"kubernetes.io/projected/7ae36837-6f06-4bfb-87b1-4bcfed69167e-kube-api-access-59tbn\") pod \"maas-api-revoke-sub-7ffc4cba-nd6dz\" (UID: \"7ae36837-6f06-4bfb-87b1-4bcfed69167e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ffc4cba-nd6dz" Nov 01 06:06:12.374731 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:12.371819 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs"] Nov 01 06:06:12.393326 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:12.390326 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59tbn\" (UniqueName: \"kubernetes.io/projected/7ae36837-6f06-4bfb-87b1-4bcfed69167e-kube-api-access-59tbn\") pod \"maas-api-revoke-sub-7ffc4cba-nd6dz\" (UID: \"7ae36837-6f06-4bfb-87b1-4bcfed69167e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ffc4cba-nd6dz" Nov 01 06:06:12.397502 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:12.395982 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ae36837-6f06-4bfb-87b1-4bcfed69167e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ffc4cba-nd6dz\" (UID: \"7ae36837-6f06-4bfb-87b1-4bcfed69167e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ffc4cba-nd6dz" Nov 01 06:06:12.397872 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:12.397843 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ae36837-6f06-4bfb-87b1-4bcfed69167e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ffc4cba-nd6dz\" (UID: \"7ae36837-6f06-4bfb-87b1-4bcfed69167e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ffc4cba-nd6dz" Nov 01 06:06:12.402425 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:12.399023 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ae36837-6f06-4bfb-87b1-4bcfed69167e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ffc4cba-nd6dz\" (UID: \"7ae36837-6f06-4bfb-87b1-4bcfed69167e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ffc4cba-nd6dz" Nov 01 06:06:12.411469 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:12.410121 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ae36837-6f06-4bfb-87b1-4bcfed69167e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ffc4cba-nd6dz\" (UID: \"7ae36837-6f06-4bfb-87b1-4bcfed69167e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ffc4cba-nd6dz" Nov 01 06:06:12.458214 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:12.458179 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59tbn\" (UniqueName: \"kubernetes.io/projected/7ae36837-6f06-4bfb-87b1-4bcfed69167e-kube-api-access-59tbn\") pod \"maas-api-revoke-sub-7ffc4cba-nd6dz\" (UID: \"7ae36837-6f06-4bfb-87b1-4bcfed69167e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ffc4cba-nd6dz" Nov 01 06:06:12.575295 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:12.575257 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ffc4cba-nd6dz" Nov 01 06:06:12.990725 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:06:12.990696 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ae36837_6f06_4bfb_87b1_4bcfed69167e.slice/crio-4fc3b5e736fcb006503c7cef11028bc79c115512df44183761b402dbf5b10ae3 WatchSource:0}: Error finding container 4fc3b5e736fcb006503c7cef11028bc79c115512df44183761b402dbf5b10ae3: Status 404 returned error can't find the container with id 4fc3b5e736fcb006503c7cef11028bc79c115512df44183761b402dbf5b10ae3 Nov 01 06:06:12.991330 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:12.991304 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ffc4cba-nd6dz"] Nov 01 06:06:13.494503 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:13.494435 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ffc4cba-nd6dz" event={"ID":"7ae36837-6f06-4bfb-87b1-4bcfed69167e","Type":"ContainerStarted","Data":"709b7d54a7725413ef448845c4dbfd191a34350d2fc69a8f17d6840e013af177"} Nov 01 06:06:13.494911 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:13.494525 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ffc4cba-nd6dz" event={"ID":"7ae36837-6f06-4bfb-87b1-4bcfed69167e","Type":"ContainerStarted","Data":"4fc3b5e736fcb006503c7cef11028bc79c115512df44183761b402dbf5b10ae3"} Nov 01 06:06:13.611884 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:13.611787 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ffc4cba-nd6dz" podStartSLOduration=2.61177078 podStartE2EDuration="2.61177078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:06:13.610874773 +0000 UTC m=+1252.265991941" watchObservedRunningTime="2026-11-01 06:06:13.61177078 +0000 UTC m=+1252.266887948" Nov 01 06:06:14.505644 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:14.505606 5040 generic.go:356] "Generic (PLEG): container finished" podID="7ae36837-6f06-4bfb-87b1-4bcfed69167e" containerID="709b7d54a7725413ef448845c4dbfd191a34350d2fc69a8f17d6840e013af177" exitCode=0 Nov 01 06:06:14.506166 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:14.505746 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ffc4cba-nd6dz" event={"ID":"7ae36837-6f06-4bfb-87b1-4bcfed69167e","Type":"ContainerDied","Data":"709b7d54a7725413ef448845c4dbfd191a34350d2fc69a8f17d6840e013af177"} Nov 01 06:06:14.921719 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:14.921636 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c0400ea-9wkhc"] Nov 01 06:06:14.925696 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:14.925674 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c0400ea-9wkhc" Nov 01 06:06:14.983002 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:14.982964 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c0400ea-9wkhc"] Nov 01 06:06:15.027282 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:15.027245 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65dbb0a6-e671-4131-945a-5a15c9433b25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c0400ea-9wkhc\" (UID: \"65dbb0a6-e671-4131-945a-5a15c9433b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c0400ea-9wkhc" Nov 01 06:06:15.027282 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:15.027282 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpkzp\" (UniqueName: \"kubernetes.io/projected/65dbb0a6-e671-4131-945a-5a15c9433b25-kube-api-access-vpkzp\") pod \"maas-api-revoke-sub-3c0400ea-9wkhc\" (UID: \"65dbb0a6-e671-4131-945a-5a15c9433b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c0400ea-9wkhc" Nov 01 06:06:15.027573 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:15.027431 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65dbb0a6-e671-4131-945a-5a15c9433b25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c0400ea-9wkhc\" (UID: \"65dbb0a6-e671-4131-945a-5a15c9433b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c0400ea-9wkhc" Nov 01 06:06:15.129070 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:15.129032 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65dbb0a6-e671-4131-945a-5a15c9433b25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c0400ea-9wkhc\" (UID: \"65dbb0a6-e671-4131-945a-5a15c9433b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c0400ea-9wkhc" Nov 01 06:06:15.129070 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:15.129072 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpkzp\" (UniqueName: \"kubernetes.io/projected/65dbb0a6-e671-4131-945a-5a15c9433b25-kube-api-access-vpkzp\") pod \"maas-api-revoke-sub-3c0400ea-9wkhc\" (UID: \"65dbb0a6-e671-4131-945a-5a15c9433b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c0400ea-9wkhc" Nov 01 06:06:15.129340 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:15.129130 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65dbb0a6-e671-4131-945a-5a15c9433b25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c0400ea-9wkhc\" (UID: \"65dbb0a6-e671-4131-945a-5a15c9433b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c0400ea-9wkhc" Nov 01 06:06:15.129981 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:15.129953 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65dbb0a6-e671-4131-945a-5a15c9433b25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c0400ea-9wkhc\" (UID: \"65dbb0a6-e671-4131-945a-5a15c9433b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c0400ea-9wkhc" Nov 01 06:06:15.131586 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:15.131562 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65dbb0a6-e671-4131-945a-5a15c9433b25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c0400ea-9wkhc\" (UID: \"65dbb0a6-e671-4131-945a-5a15c9433b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c0400ea-9wkhc" Nov 01 06:06:15.191183 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:15.191106 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpkzp\" (UniqueName: \"kubernetes.io/projected/65dbb0a6-e671-4131-945a-5a15c9433b25-kube-api-access-vpkzp\") pod \"maas-api-revoke-sub-3c0400ea-9wkhc\" (UID: \"65dbb0a6-e671-4131-945a-5a15c9433b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c0400ea-9wkhc" Nov 01 06:06:15.239135 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:15.239098 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c0400ea-9wkhc" Nov 01 06:06:15.624063 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:06:15.624034 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65dbb0a6_e671_4131_945a_5a15c9433b25.slice/crio-eea5c49f714ad0f6d7246e3c0dfba56493e64dbf56cc122ea565ea5a95dfe998 WatchSource:0}: Error finding container eea5c49f714ad0f6d7246e3c0dfba56493e64dbf56cc122ea565ea5a95dfe998: Status 404 returned error can't find the container with id eea5c49f714ad0f6d7246e3c0dfba56493e64dbf56cc122ea565ea5a95dfe998 Nov 01 06:06:15.628351 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:15.628320 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c0400ea-9wkhc"] Nov 01 06:06:15.674934 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:15.674910 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ffc4cba-nd6dz" Nov 01 06:06:15.735915 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:15.735878 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d52e3446-bprl4"] Nov 01 06:06:15.736474 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:15.736455 5040 cpu_manager.go:414] "containerMap: removing container" podUID="7ae36837-6f06-4bfb-87b1-4bcfed69167e" containerName="revoke-keys" Nov 01 06:06:15.736474 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:15.736488 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ae36837-6f06-4bfb-87b1-4bcfed69167e" containerName="revoke-keys" Nov 01 06:06:15.736662 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:15.736649 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ae36837-6f06-4bfb-87b1-4bcfed69167e" containerName="revoke-keys" Nov 01 06:06:15.740080 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:15.740061 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52e3446-bprl4" Nov 01 06:06:15.804449 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:15.804364 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d52e3446-bprl4"] Nov 01 06:06:15.836589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:15.836557 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ae36837-6f06-4bfb-87b1-4bcfed69167e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ae36837-6f06-4bfb-87b1-4bcfed69167e-maas-api-openshift-ca\") pod \"7ae36837-6f06-4bfb-87b1-4bcfed69167e\" (UID: \"7ae36837-6f06-4bfb-87b1-4bcfed69167e\") " Nov 01 06:06:15.836589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:15.836600 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ae36837-6f06-4bfb-87b1-4bcfed69167e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ae36837-6f06-4bfb-87b1-4bcfed69167e-maas-api-xks-ca\") pod \"7ae36837-6f06-4bfb-87b1-4bcfed69167e\" (UID: \"7ae36837-6f06-4bfb-87b1-4bcfed69167e\") " Nov 01 06:06:15.836805 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:15.836739 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ae36837-6f06-4bfb-87b1-4bcfed69167e-kube-api-access-59tbn\" (UniqueName: \"kubernetes.io/projected/7ae36837-6f06-4bfb-87b1-4bcfed69167e-kube-api-access-59tbn\") pod \"7ae36837-6f06-4bfb-87b1-4bcfed69167e\" (UID: \"7ae36837-6f06-4bfb-87b1-4bcfed69167e\") " Nov 01 06:06:15.836894 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:15.836874 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjmww\" (UniqueName: \"kubernetes.io/projected/c10af860-64f5-4bc5-bb86-93b48de1a0f3-kube-api-access-vjmww\") pod \"maas-api-revoke-sub-d52e3446-bprl4\" (UID: \"c10af860-64f5-4bc5-bb86-93b48de1a0f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52e3446-bprl4" Nov 01 06:06:15.836955 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:15.836912 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c10af860-64f5-4bc5-bb86-93b48de1a0f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d52e3446-bprl4\" (UID: \"c10af860-64f5-4bc5-bb86-93b48de1a0f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52e3446-bprl4" Nov 01 06:06:15.837011 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:15.836990 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ae36837-6f06-4bfb-87b1-4bcfed69167e-maas-api-openshift-ca" pod "7ae36837-6f06-4bfb-87b1-4bcfed69167e" (UID: "7ae36837-6f06-4bfb-87b1-4bcfed69167e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:06:15.837155 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:15.837136 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c10af860-64f5-4bc5-bb86-93b48de1a0f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d52e3446-bprl4\" (UID: \"c10af860-64f5-4bc5-bb86-93b48de1a0f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52e3446-bprl4" Nov 01 06:06:15.837266 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:15.837251 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ae36837-6f06-4bfb-87b1-4bcfed69167e-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:15.838995 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:15.838973 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ae36837-6f06-4bfb-87b1-4bcfed69167e-maas-api-xks-ca" pod "7ae36837-6f06-4bfb-87b1-4bcfed69167e" (UID: "7ae36837-6f06-4bfb-87b1-4bcfed69167e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:06:15.839078 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:15.838994 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ae36837-6f06-4bfb-87b1-4bcfed69167e-kube-api-access-59tbn" pod "7ae36837-6f06-4bfb-87b1-4bcfed69167e" (UID: "7ae36837-6f06-4bfb-87b1-4bcfed69167e"). InnerVolumeSpecName "kube-api-access-59tbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:06:15.938474 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:15.938435 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjmww\" (UniqueName: \"kubernetes.io/projected/c10af860-64f5-4bc5-bb86-93b48de1a0f3-kube-api-access-vjmww\") pod \"maas-api-revoke-sub-d52e3446-bprl4\" (UID: \"c10af860-64f5-4bc5-bb86-93b48de1a0f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52e3446-bprl4" Nov 01 06:06:15.938682 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:15.938508 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c10af860-64f5-4bc5-bb86-93b48de1a0f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d52e3446-bprl4\" (UID: \"c10af860-64f5-4bc5-bb86-93b48de1a0f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52e3446-bprl4" Nov 01 06:06:15.938734 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:15.938716 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c10af860-64f5-4bc5-bb86-93b48de1a0f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d52e3446-bprl4\" (UID: \"c10af860-64f5-4bc5-bb86-93b48de1a0f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52e3446-bprl4" Nov 01 06:06:15.938819 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:15.938805 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ae36837-6f06-4bfb-87b1-4bcfed69167e-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:15.938861 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:15.938825 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59tbn\" (UniqueName: \"kubernetes.io/projected/7ae36837-6f06-4bfb-87b1-4bcfed69167e-kube-api-access-59tbn\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:15.939317 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:15.939292 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c10af860-64f5-4bc5-bb86-93b48de1a0f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d52e3446-bprl4\" (UID: \"c10af860-64f5-4bc5-bb86-93b48de1a0f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52e3446-bprl4" Nov 01 06:06:15.940906 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:15.940884 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c10af860-64f5-4bc5-bb86-93b48de1a0f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d52e3446-bprl4\" (UID: \"c10af860-64f5-4bc5-bb86-93b48de1a0f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52e3446-bprl4" Nov 01 06:06:16.000203 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:16.000162 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjmww\" (UniqueName: \"kubernetes.io/projected/c10af860-64f5-4bc5-bb86-93b48de1a0f3-kube-api-access-vjmww\") pod \"maas-api-revoke-sub-d52e3446-bprl4\" (UID: \"c10af860-64f5-4bc5-bb86-93b48de1a0f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52e3446-bprl4" Nov 01 06:06:16.053632 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:16.053600 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52e3446-bprl4" Nov 01 06:06:16.429923 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:06:16.429885 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc10af860_64f5_4bc5_bb86_93b48de1a0f3.slice/crio-b3da488640efccc24aa9cd91ac6169b81e960472514fcd727642e7433bf0b179 WatchSource:0}: Error finding container b3da488640efccc24aa9cd91ac6169b81e960472514fcd727642e7433bf0b179: Status 404 returned error can't find the container with id b3da488640efccc24aa9cd91ac6169b81e960472514fcd727642e7433bf0b179 Nov 01 06:06:16.430022 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:16.429999 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d52e3446-bprl4"] Nov 01 06:06:16.521290 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:16.521251 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52e3446-bprl4" event={"ID":"c10af860-64f5-4bc5-bb86-93b48de1a0f3","Type":"ContainerStarted","Data":"b3da488640efccc24aa9cd91ac6169b81e960472514fcd727642e7433bf0b179"} Nov 01 06:06:16.522808 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:16.522779 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c0400ea-9wkhc" event={"ID":"65dbb0a6-e671-4131-945a-5a15c9433b25","Type":"ContainerStarted","Data":"28e8d7041c573bab3e7da6734f6dd1e52720cf936b546534a7bcfd7198bf481b"} Nov 01 06:06:16.522964 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:16.522826 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c0400ea-9wkhc" event={"ID":"65dbb0a6-e671-4131-945a-5a15c9433b25","Type":"ContainerStarted","Data":"eea5c49f714ad0f6d7246e3c0dfba56493e64dbf56cc122ea565ea5a95dfe998"} Nov 01 06:06:16.524291 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:16.524262 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ffc4cba-nd6dz" Nov 01 06:06:16.524393 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:16.524272 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ffc4cba-nd6dz" event={"ID":"7ae36837-6f06-4bfb-87b1-4bcfed69167e","Type":"ContainerDied","Data":"4fc3b5e736fcb006503c7cef11028bc79c115512df44183761b402dbf5b10ae3"} Nov 01 06:06:16.524432 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:16.524391 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fc3b5e736fcb006503c7cef11028bc79c115512df44183761b402dbf5b10ae3" Nov 01 06:06:16.639741 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:16.639694 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c0400ea-9wkhc" podStartSLOduration=2.639680892 podStartE2EDuration="2.639680892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:06:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:06:16.638671054 +0000 UTC m=+1255.293788221" watchObservedRunningTime="2026-11-01 06:06:16.639680892 +0000 UTC m=+1255.294798296" Nov 01 06:06:17.373101 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:17.373042 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs" podUID="0da63cdc-8890-49d3-b6f5-ba7396bdc309" containerName="payload-processing" containerID="cri-o://1493964669af1a9c31e3ae4158bf1a5f29ca785226db702c6f3d27932db35038" gracePeriod=25 Nov 01 06:06:17.535132 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:17.535053 5040 generic.go:356] "Generic (PLEG): container finished" podID="0da63cdc-8890-49d3-b6f5-ba7396bdc309" containerID="1493964669af1a9c31e3ae4158bf1a5f29ca785226db702c6f3d27932db35038" exitCode=0 Nov 01 06:06:17.535293 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:17.535125 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs" event={"ID":"0da63cdc-8890-49d3-b6f5-ba7396bdc309","Type":"ContainerDied","Data":"1493964669af1a9c31e3ae4158bf1a5f29ca785226db702c6f3d27932db35038"} Nov 01 06:06:17.537012 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:17.536984 5040 generic.go:356] "Generic (PLEG): container finished" podID="c10af860-64f5-4bc5-bb86-93b48de1a0f3" containerID="32fdafba2fa6ab689859aa6ef413ba3679ddce74b72ca568228b9a34e5586d20" exitCode=0 Nov 01 06:06:17.537130 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:17.537071 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52e3446-bprl4" event={"ID":"c10af860-64f5-4bc5-bb86-93b48de1a0f3","Type":"ContainerDied","Data":"32fdafba2fa6ab689859aa6ef413ba3679ddce74b72ca568228b9a34e5586d20"} Nov 01 06:06:17.540036 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:17.540013 5040 generic.go:356] "Generic (PLEG): container finished" podID="65dbb0a6-e671-4131-945a-5a15c9433b25" containerID="28e8d7041c573bab3e7da6734f6dd1e52720cf936b546534a7bcfd7198bf481b" exitCode=0 Nov 01 06:06:17.540210 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:17.540196 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c0400ea-9wkhc" event={"ID":"65dbb0a6-e671-4131-945a-5a15c9433b25","Type":"ContainerDied","Data":"28e8d7041c573bab3e7da6734f6dd1e52720cf936b546534a7bcfd7198bf481b"} Nov 01 06:06:17.668006 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:17.667976 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs" Nov 01 06:06:17.759227 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:17.759195 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0da63cdc-8890-49d3-b6f5-ba7396bdc309-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0da63cdc-8890-49d3-b6f5-ba7396bdc309-routing-overlay-volume\") pod \"0da63cdc-8890-49d3-b6f5-ba7396bdc309\" (UID: \"0da63cdc-8890-49d3-b6f5-ba7396bdc309\") " Nov 01 06:06:17.759396 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:17.759247 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0da63cdc-8890-49d3-b6f5-ba7396bdc309-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0da63cdc-8890-49d3-b6f5-ba7396bdc309-plugins-config-volume\") pod \"0da63cdc-8890-49d3-b6f5-ba7396bdc309\" (UID: \"0da63cdc-8890-49d3-b6f5-ba7396bdc309\") " Nov 01 06:06:17.759396 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:17.759278 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da63cdc-8890-49d3-b6f5-ba7396bdc309-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0da63cdc-8890-49d3-b6f5-ba7396bdc309-provider-credentials\") pod \"0da63cdc-8890-49d3-b6f5-ba7396bdc309\" (UID: \"0da63cdc-8890-49d3-b6f5-ba7396bdc309\") " Nov 01 06:06:17.759538 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:17.759430 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da63cdc-8890-49d3-b6f5-ba7396bdc309-kube-api-access-sbhmj\" (UniqueName: \"kubernetes.io/projected/0da63cdc-8890-49d3-b6f5-ba7396bdc309-kube-api-access-sbhmj\") pod \"0da63cdc-8890-49d3-b6f5-ba7396bdc309\" (UID: \"0da63cdc-8890-49d3-b6f5-ba7396bdc309\") " Nov 01 06:06:17.759680 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:17.759650 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da63cdc-8890-49d3-b6f5-ba7396bdc309-routing-overlay-volume" pod "0da63cdc-8890-49d3-b6f5-ba7396bdc309" (UID: "0da63cdc-8890-49d3-b6f5-ba7396bdc309"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:06:17.759746 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:17.759705 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da63cdc-8890-49d3-b6f5-ba7396bdc309-plugins-config-volume" pod "0da63cdc-8890-49d3-b6f5-ba7396bdc309" (UID: "0da63cdc-8890-49d3-b6f5-ba7396bdc309"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:06:17.759805 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:17.759785 5040 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0da63cdc-8890-49d3-b6f5-ba7396bdc309-routing-overlay-volume\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:17.761780 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:17.761748 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da63cdc-8890-49d3-b6f5-ba7396bdc309-kube-api-access-sbhmj" pod "0da63cdc-8890-49d3-b6f5-ba7396bdc309" (UID: "0da63cdc-8890-49d3-b6f5-ba7396bdc309"). InnerVolumeSpecName "kube-api-access-sbhmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:06:17.761907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:17.761756 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da63cdc-8890-49d3-b6f5-ba7396bdc309-provider-credentials" pod "0da63cdc-8890-49d3-b6f5-ba7396bdc309" (UID: "0da63cdc-8890-49d3-b6f5-ba7396bdc309"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:06:17.860868 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:17.860783 5040 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0da63cdc-8890-49d3-b6f5-ba7396bdc309-plugins-config-volume\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:17.860868 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:17.860814 5040 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0da63cdc-8890-49d3-b6f5-ba7396bdc309-provider-credentials\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:17.860868 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:17.860824 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbhmj\" (UniqueName: \"kubernetes.io/projected/0da63cdc-8890-49d3-b6f5-ba7396bdc309-kube-api-access-sbhmj\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:18.550136 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:18.550097 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs" event={"ID":"0da63cdc-8890-49d3-b6f5-ba7396bdc309","Type":"ContainerDied","Data":"372729aec3be735097534bb75abb8e1feac6a48eb4077ee952476bc6bea5c02d"} Nov 01 06:06:18.550136 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:18.550145 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs" Nov 01 06:06:18.550361 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:18.550175 5040 scope.go:122] "RemoveContainer" containerID="1493964669af1a9c31e3ae4158bf1a5f29ca785226db702c6f3d27932db35038" Nov 01 06:06:18.722810 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:18.722777 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs"] Nov 01 06:06:18.768887 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:18.768855 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52e3446-bprl4" Nov 01 06:06:18.772035 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:18.772015 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c0400ea-9wkhc" Nov 01 06:06:18.772922 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:18.772901 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7ddjvvs"] Nov 01 06:06:18.873842 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:18.873734 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65dbb0a6-e671-4131-945a-5a15c9433b25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65dbb0a6-e671-4131-945a-5a15c9433b25-maas-api-openshift-ca\") pod \"65dbb0a6-e671-4131-945a-5a15c9433b25\" (UID: \"65dbb0a6-e671-4131-945a-5a15c9433b25\") " Nov 01 06:06:18.873842 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:18.873796 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65dbb0a6-e671-4131-945a-5a15c9433b25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65dbb0a6-e671-4131-945a-5a15c9433b25-maas-api-xks-ca\") pod \"65dbb0a6-e671-4131-945a-5a15c9433b25\" (UID: \"65dbb0a6-e671-4131-945a-5a15c9433b25\") " Nov 01 06:06:18.873842 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:18.873842 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c10af860-64f5-4bc5-bb86-93b48de1a0f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c10af860-64f5-4bc5-bb86-93b48de1a0f3-maas-api-openshift-ca\") pod \"c10af860-64f5-4bc5-bb86-93b48de1a0f3\" (UID: \"c10af860-64f5-4bc5-bb86-93b48de1a0f3\") " Nov 01 06:06:18.874116 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:18.873890 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c10af860-64f5-4bc5-bb86-93b48de1a0f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c10af860-64f5-4bc5-bb86-93b48de1a0f3-maas-api-xks-ca\") pod \"c10af860-64f5-4bc5-bb86-93b48de1a0f3\" (UID: \"c10af860-64f5-4bc5-bb86-93b48de1a0f3\") " Nov 01 06:06:18.874116 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:18.873922 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65dbb0a6-e671-4131-945a-5a15c9433b25-kube-api-access-vpkzp\" (UniqueName: \"kubernetes.io/projected/65dbb0a6-e671-4131-945a-5a15c9433b25-kube-api-access-vpkzp\") pod \"65dbb0a6-e671-4131-945a-5a15c9433b25\" (UID: \"65dbb0a6-e671-4131-945a-5a15c9433b25\") " Nov 01 06:06:18.874116 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:18.873967 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c10af860-64f5-4bc5-bb86-93b48de1a0f3-kube-api-access-vjmww\" (UniqueName: \"kubernetes.io/projected/c10af860-64f5-4bc5-bb86-93b48de1a0f3-kube-api-access-vjmww\") pod \"c10af860-64f5-4bc5-bb86-93b48de1a0f3\" (UID: \"c10af860-64f5-4bc5-bb86-93b48de1a0f3\") " Nov 01 06:06:18.874269 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:18.874170 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65dbb0a6-e671-4131-945a-5a15c9433b25-maas-api-openshift-ca" pod "65dbb0a6-e671-4131-945a-5a15c9433b25" (UID: "65dbb0a6-e671-4131-945a-5a15c9433b25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:06:18.874343 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:18.874318 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c10af860-64f5-4bc5-bb86-93b48de1a0f3-maas-api-openshift-ca" pod "c10af860-64f5-4bc5-bb86-93b48de1a0f3" (UID: "c10af860-64f5-4bc5-bb86-93b48de1a0f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:06:18.874600 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:18.874576 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65dbb0a6-e671-4131-945a-5a15c9433b25-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:18.874718 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:18.874604 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c10af860-64f5-4bc5-bb86-93b48de1a0f3-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:18.876141 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:18.876114 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65dbb0a6-e671-4131-945a-5a15c9433b25-kube-api-access-vpkzp" pod "65dbb0a6-e671-4131-945a-5a15c9433b25" (UID: "65dbb0a6-e671-4131-945a-5a15c9433b25"). InnerVolumeSpecName "kube-api-access-vpkzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:06:18.876237 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:18.876213 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c10af860-64f5-4bc5-bb86-93b48de1a0f3-kube-api-access-vjmww" pod "c10af860-64f5-4bc5-bb86-93b48de1a0f3" (UID: "c10af860-64f5-4bc5-bb86-93b48de1a0f3"). InnerVolumeSpecName "kube-api-access-vjmww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:06:18.876413 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:18.876389 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65dbb0a6-e671-4131-945a-5a15c9433b25-maas-api-xks-ca" pod "65dbb0a6-e671-4131-945a-5a15c9433b25" (UID: "65dbb0a6-e671-4131-945a-5a15c9433b25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:06:18.876525 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:18.876415 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c10af860-64f5-4bc5-bb86-93b48de1a0f3-maas-api-xks-ca" pod "c10af860-64f5-4bc5-bb86-93b48de1a0f3" (UID: "c10af860-64f5-4bc5-bb86-93b48de1a0f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:06:18.976042 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:18.976005 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65dbb0a6-e671-4131-945a-5a15c9433b25-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:18.976042 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:18.976037 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c10af860-64f5-4bc5-bb86-93b48de1a0f3-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:18.976042 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:18.976047 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpkzp\" (UniqueName: \"kubernetes.io/projected/65dbb0a6-e671-4131-945a-5a15c9433b25-kube-api-access-vpkzp\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:18.976291 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:18.976057 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjmww\" (UniqueName: \"kubernetes.io/projected/c10af860-64f5-4bc5-bb86-93b48de1a0f3-kube-api-access-vjmww\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:19.515529 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:19.515493 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0da63cdc-8890-49d3-b6f5-ba7396bdc309" path="/var/lib/kubelet/pods/0da63cdc-8890-49d3-b6f5-ba7396bdc309/volumes" Nov 01 06:06:19.559950 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:19.559923 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52e3446-bprl4" Nov 01 06:06:19.560142 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:19.559916 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52e3446-bprl4" event={"ID":"c10af860-64f5-4bc5-bb86-93b48de1a0f3","Type":"ContainerDied","Data":"b3da488640efccc24aa9cd91ac6169b81e960472514fcd727642e7433bf0b179"} Nov 01 06:06:19.560142 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:19.560053 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3da488640efccc24aa9cd91ac6169b81e960472514fcd727642e7433bf0b179" Nov 01 06:06:19.561429 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:19.561411 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c0400ea-9wkhc" Nov 01 06:06:19.561551 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:19.561451 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c0400ea-9wkhc" event={"ID":"65dbb0a6-e671-4131-945a-5a15c9433b25","Type":"ContainerDied","Data":"eea5c49f714ad0f6d7246e3c0dfba56493e64dbf56cc122ea565ea5a95dfe998"} Nov 01 06:06:19.561551 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:19.561504 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eea5c49f714ad0f6d7246e3c0dfba56493e64dbf56cc122ea565ea5a95dfe998" Nov 01 06:06:26.485504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:26.485450 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf6b940e-nhpmt"] Nov 01 06:06:26.485962 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:26.485949 5040 cpu_manager.go:414] "containerMap: removing container" podUID="0da63cdc-8890-49d3-b6f5-ba7396bdc309" containerName="payload-processing" Nov 01 06:06:26.486023 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:26.485963 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0da63cdc-8890-49d3-b6f5-ba7396bdc309" containerName="payload-processing" Nov 01 06:06:26.486023 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:26.485978 5040 cpu_manager.go:414] "containerMap: removing container" podUID="65dbb0a6-e671-4131-945a-5a15c9433b25" containerName="revoke-keys" Nov 01 06:06:26.486023 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:26.485984 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65dbb0a6-e671-4131-945a-5a15c9433b25" containerName="revoke-keys" Nov 01 06:06:26.486023 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:26.486009 5040 cpu_manager.go:414] "containerMap: removing container" podUID="c10af860-64f5-4bc5-bb86-93b48de1a0f3" containerName="revoke-keys" Nov 01 06:06:26.486023 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:26.486015 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c10af860-64f5-4bc5-bb86-93b48de1a0f3" containerName="revoke-keys" Nov 01 06:06:26.486169 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:26.486093 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="c10af860-64f5-4bc5-bb86-93b48de1a0f3" containerName="revoke-keys" Nov 01 06:06:26.486169 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:26.486104 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="0da63cdc-8890-49d3-b6f5-ba7396bdc309" containerName="payload-processing" Nov 01 06:06:26.486169 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:26.486111 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="65dbb0a6-e671-4131-945a-5a15c9433b25" containerName="revoke-keys" Nov 01 06:06:26.490908 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:26.490880 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6b940e-nhpmt" Nov 01 06:06:26.552889 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:26.552849 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf6b940e-nhpmt"] Nov 01 06:06:26.557727 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:26.557696 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:06:26.557897 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:26.557783 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:06:26.558368 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:26.558344 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:06:26.558662 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:26.558568 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:06:26.569073 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:26.569037 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnggg\" (UniqueName: \"kubernetes.io/projected/998ef654-d73e-4a90-9e6a-f45d90cc959c-kube-api-access-cnggg\") pod \"maas-api-revoke-sub-bf6b940e-nhpmt\" (UID: \"998ef654-d73e-4a90-9e6a-f45d90cc959c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6b940e-nhpmt" Nov 01 06:06:26.569304 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:26.569283 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/998ef654-d73e-4a90-9e6a-f45d90cc959c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf6b940e-nhpmt\" (UID: \"998ef654-d73e-4a90-9e6a-f45d90cc959c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6b940e-nhpmt" Nov 01 06:06:26.569602 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:26.569580 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/998ef654-d73e-4a90-9e6a-f45d90cc959c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf6b940e-nhpmt\" (UID: \"998ef654-d73e-4a90-9e6a-f45d90cc959c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6b940e-nhpmt" Nov 01 06:06:26.670352 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:26.670320 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/998ef654-d73e-4a90-9e6a-f45d90cc959c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf6b940e-nhpmt\" (UID: \"998ef654-d73e-4a90-9e6a-f45d90cc959c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6b940e-nhpmt" Nov 01 06:06:26.670561 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:26.670431 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnggg\" (UniqueName: \"kubernetes.io/projected/998ef654-d73e-4a90-9e6a-f45d90cc959c-kube-api-access-cnggg\") pod \"maas-api-revoke-sub-bf6b940e-nhpmt\" (UID: \"998ef654-d73e-4a90-9e6a-f45d90cc959c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6b940e-nhpmt" Nov 01 06:06:26.670561 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:26.670456 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/998ef654-d73e-4a90-9e6a-f45d90cc959c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf6b940e-nhpmt\" (UID: \"998ef654-d73e-4a90-9e6a-f45d90cc959c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6b940e-nhpmt" Nov 01 06:06:26.671006 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:26.670983 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/998ef654-d73e-4a90-9e6a-f45d90cc959c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf6b940e-nhpmt\" (UID: \"998ef654-d73e-4a90-9e6a-f45d90cc959c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6b940e-nhpmt" Nov 01 06:06:26.672727 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:26.672704 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/998ef654-d73e-4a90-9e6a-f45d90cc959c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf6b940e-nhpmt\" (UID: \"998ef654-d73e-4a90-9e6a-f45d90cc959c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6b940e-nhpmt" Nov 01 06:06:26.731988 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:26.731956 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnggg\" (UniqueName: \"kubernetes.io/projected/998ef654-d73e-4a90-9e6a-f45d90cc959c-kube-api-access-cnggg\") pod \"maas-api-revoke-sub-bf6b940e-nhpmt\" (UID: \"998ef654-d73e-4a90-9e6a-f45d90cc959c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6b940e-nhpmt" Nov 01 06:06:26.804965 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:26.804935 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6b940e-nhpmt" Nov 01 06:06:27.181301 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:06:27.181268 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod998ef654_d73e_4a90_9e6a_f45d90cc959c.slice/crio-cfc391cbb4ebd0f05496a5945061a9c3fc00573db1795bac982834c584ee0188 WatchSource:0}: Error finding container cfc391cbb4ebd0f05496a5945061a9c3fc00573db1795bac982834c584ee0188: Status 404 returned error can't find the container with id cfc391cbb4ebd0f05496a5945061a9c3fc00573db1795bac982834c584ee0188 Nov 01 06:06:27.181771 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:27.181745 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf6b940e-nhpmt"] Nov 01 06:06:27.623132 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:27.623100 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6b940e-nhpmt" event={"ID":"998ef654-d73e-4a90-9e6a-f45d90cc959c","Type":"ContainerStarted","Data":"6a8c2d06c7ff9e57a2a8290d656f6c152ed57372ced5fa4529b4d10c62779d6a"} Nov 01 06:06:27.623617 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:27.623150 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6b940e-nhpmt" event={"ID":"998ef654-d73e-4a90-9e6a-f45d90cc959c","Type":"ContainerStarted","Data":"cfc391cbb4ebd0f05496a5945061a9c3fc00573db1795bac982834c584ee0188"} Nov 01 06:06:27.740228 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:27.740173 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6b940e-nhpmt" podStartSLOduration=1.740159638 podStartE2EDuration="1.740159638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:06:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:06:27.7394528 +0000 UTC m=+1266.394569967" watchObservedRunningTime="2026-11-01 06:06:27.740159638 +0000 UTC m=+1266.395276805" Nov 01 06:06:28.632459 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:28.632423 5040 generic.go:356] "Generic (PLEG): container finished" podID="998ef654-d73e-4a90-9e6a-f45d90cc959c" containerID="6a8c2d06c7ff9e57a2a8290d656f6c152ed57372ced5fa4529b4d10c62779d6a" exitCode=0 Nov 01 06:06:28.632914 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:28.632513 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6b940e-nhpmt" event={"ID":"998ef654-d73e-4a90-9e6a-f45d90cc959c","Type":"ContainerDied","Data":"6a8c2d06c7ff9e57a2a8290d656f6c152ed57372ced5fa4529b4d10c62779d6a"} Nov 01 06:06:29.369654 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:29.369620 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46c05f3b-r78db"] Nov 01 06:06:29.373694 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:29.373676 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c05f3b-r78db" Nov 01 06:06:29.401203 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:29.401173 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852e1076-47cc-4f39-bc85-88719ff826a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46c05f3b-r78db\" (UID: \"852e1076-47cc-4f39-bc85-88719ff826a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c05f3b-r78db" Nov 01 06:06:29.401340 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:29.401235 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zljfg\" (UniqueName: \"kubernetes.io/projected/852e1076-47cc-4f39-bc85-88719ff826a5-kube-api-access-zljfg\") pod \"maas-api-revoke-sub-46c05f3b-r78db\" (UID: \"852e1076-47cc-4f39-bc85-88719ff826a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c05f3b-r78db" Nov 01 06:06:29.401340 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:29.401323 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852e1076-47cc-4f39-bc85-88719ff826a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46c05f3b-r78db\" (UID: \"852e1076-47cc-4f39-bc85-88719ff826a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c05f3b-r78db" Nov 01 06:06:29.430866 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:29.430835 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46c05f3b-r78db"] Nov 01 06:06:29.502113 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:29.502073 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852e1076-47cc-4f39-bc85-88719ff826a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46c05f3b-r78db\" (UID: \"852e1076-47cc-4f39-bc85-88719ff826a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c05f3b-r78db" Nov 01 06:06:29.502311 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:29.502226 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zljfg\" (UniqueName: \"kubernetes.io/projected/852e1076-47cc-4f39-bc85-88719ff826a5-kube-api-access-zljfg\") pod \"maas-api-revoke-sub-46c05f3b-r78db\" (UID: \"852e1076-47cc-4f39-bc85-88719ff826a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c05f3b-r78db" Nov 01 06:06:29.502311 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:29.502277 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852e1076-47cc-4f39-bc85-88719ff826a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46c05f3b-r78db\" (UID: \"852e1076-47cc-4f39-bc85-88719ff826a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c05f3b-r78db" Nov 01 06:06:29.502972 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:29.502947 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852e1076-47cc-4f39-bc85-88719ff826a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46c05f3b-r78db\" (UID: \"852e1076-47cc-4f39-bc85-88719ff826a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c05f3b-r78db" Nov 01 06:06:29.504570 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:29.504548 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852e1076-47cc-4f39-bc85-88719ff826a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46c05f3b-r78db\" (UID: \"852e1076-47cc-4f39-bc85-88719ff826a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c05f3b-r78db" Nov 01 06:06:29.563825 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:29.563792 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zljfg\" (UniqueName: \"kubernetes.io/projected/852e1076-47cc-4f39-bc85-88719ff826a5-kube-api-access-zljfg\") pod \"maas-api-revoke-sub-46c05f3b-r78db\" (UID: \"852e1076-47cc-4f39-bc85-88719ff826a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c05f3b-r78db" Nov 01 06:06:29.687314 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:29.687275 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c05f3b-r78db" Nov 01 06:06:29.807262 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:29.807238 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6b940e-nhpmt" Nov 01 06:06:29.907582 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:29.907474 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/998ef654-d73e-4a90-9e6a-f45d90cc959c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/998ef654-d73e-4a90-9e6a-f45d90cc959c-maas-api-xks-ca\") pod \"998ef654-d73e-4a90-9e6a-f45d90cc959c\" (UID: \"998ef654-d73e-4a90-9e6a-f45d90cc959c\") " Nov 01 06:06:29.907729 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:29.907580 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/998ef654-d73e-4a90-9e6a-f45d90cc959c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/998ef654-d73e-4a90-9e6a-f45d90cc959c-maas-api-openshift-ca\") pod \"998ef654-d73e-4a90-9e6a-f45d90cc959c\" (UID: \"998ef654-d73e-4a90-9e6a-f45d90cc959c\") " Nov 01 06:06:29.907777 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:29.907760 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/998ef654-d73e-4a90-9e6a-f45d90cc959c-kube-api-access-cnggg\" (UniqueName: \"kubernetes.io/projected/998ef654-d73e-4a90-9e6a-f45d90cc959c-kube-api-access-cnggg\") pod \"998ef654-d73e-4a90-9e6a-f45d90cc959c\" (UID: \"998ef654-d73e-4a90-9e6a-f45d90cc959c\") " Nov 01 06:06:29.907995 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:29.907967 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/998ef654-d73e-4a90-9e6a-f45d90cc959c-maas-api-openshift-ca" pod "998ef654-d73e-4a90-9e6a-f45d90cc959c" (UID: "998ef654-d73e-4a90-9e6a-f45d90cc959c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:06:29.908164 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:29.908146 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/998ef654-d73e-4a90-9e6a-f45d90cc959c-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:29.909552 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:29.909535 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/998ef654-d73e-4a90-9e6a-f45d90cc959c-maas-api-xks-ca" pod "998ef654-d73e-4a90-9e6a-f45d90cc959c" (UID: "998ef654-d73e-4a90-9e6a-f45d90cc959c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:06:29.909763 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:29.909742 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998ef654-d73e-4a90-9e6a-f45d90cc959c-kube-api-access-cnggg" pod "998ef654-d73e-4a90-9e6a-f45d90cc959c" (UID: "998ef654-d73e-4a90-9e6a-f45d90cc959c"). InnerVolumeSpecName "kube-api-access-cnggg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:06:30.009018 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:30.008985 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnggg\" (UniqueName: \"kubernetes.io/projected/998ef654-d73e-4a90-9e6a-f45d90cc959c-kube-api-access-cnggg\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:30.009018 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:30.009017 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/998ef654-d73e-4a90-9e6a-f45d90cc959c-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:30.080040 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:06:30.080008 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod852e1076_47cc_4f39_bc85_88719ff826a5.slice/crio-dd67e28d1b9d40637982145d46829c5f1fff439267ed77da53a5cf3744274469 WatchSource:0}: Error finding container dd67e28d1b9d40637982145d46829c5f1fff439267ed77da53a5cf3744274469: Status 404 returned error can't find the container with id dd67e28d1b9d40637982145d46829c5f1fff439267ed77da53a5cf3744274469 Nov 01 06:06:30.080664 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:30.080639 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46c05f3b-r78db"] Nov 01 06:06:30.651950 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:30.651847 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c05f3b-r78db" event={"ID":"852e1076-47cc-4f39-bc85-88719ff826a5","Type":"ContainerStarted","Data":"51e200a0419a44ff0527b12a1a4f82d566c61dcdafa52e468d3a1039328f59cf"} Nov 01 06:06:30.651950 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:30.651909 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c05f3b-r78db" event={"ID":"852e1076-47cc-4f39-bc85-88719ff826a5","Type":"ContainerStarted","Data":"dd67e28d1b9d40637982145d46829c5f1fff439267ed77da53a5cf3744274469"} Nov 01 06:06:30.653201 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:30.653181 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6b940e-nhpmt" Nov 01 06:06:30.653312 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:30.653215 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6b940e-nhpmt" event={"ID":"998ef654-d73e-4a90-9e6a-f45d90cc959c","Type":"ContainerDied","Data":"cfc391cbb4ebd0f05496a5945061a9c3fc00573db1795bac982834c584ee0188"} Nov 01 06:06:30.653312 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:30.653250 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfc391cbb4ebd0f05496a5945061a9c3fc00573db1795bac982834c584ee0188" Nov 01 06:06:30.769006 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:30.768953 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c05f3b-r78db" podStartSLOduration=1.76893977 podStartE2EDuration="1.76893977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:06:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:06:30.767946716 +0000 UTC m=+1269.423063895" watchObservedRunningTime="2026-11-01 06:06:30.76893977 +0000 UTC m=+1269.424056937" Nov 01 06:06:31.666214 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:31.666180 5040 generic.go:356] "Generic (PLEG): container finished" podID="852e1076-47cc-4f39-bc85-88719ff826a5" containerID="51e200a0419a44ff0527b12a1a4f82d566c61dcdafa52e468d3a1039328f59cf" exitCode=0 Nov 01 06:06:31.666402 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:31.666271 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c05f3b-r78db" event={"ID":"852e1076-47cc-4f39-bc85-88719ff826a5","Type":"ContainerDied","Data":"51e200a0419a44ff0527b12a1a4f82d566c61dcdafa52e468d3a1039328f59cf"} Nov 01 06:06:33.982567 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:33.978172 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" podUID="adbac784-4ded-4ca3-b985-3e59983fc637" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:06:34.052602 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:34.052561 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" podUID="21f54836-d1ed-4f87-8f26-7c2d939b2590" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:06:34.069667 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:34.069612 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" podUID="1878b6fb-f8fe-4f4f-8f4e-056c66fc206a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:06:34.219016 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:34.218985 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c05f3b-r78db" Nov 01 06:06:34.243548 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:34.243432 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/852e1076-47cc-4f39-bc85-88719ff826a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852e1076-47cc-4f39-bc85-88719ff826a5-maas-api-xks-ca\") pod \"852e1076-47cc-4f39-bc85-88719ff826a5\" (UID: \"852e1076-47cc-4f39-bc85-88719ff826a5\") " Nov 01 06:06:34.243874 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:34.243850 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/852e1076-47cc-4f39-bc85-88719ff826a5-kube-api-access-zljfg\" (UniqueName: \"kubernetes.io/projected/852e1076-47cc-4f39-bc85-88719ff826a5-kube-api-access-zljfg\") pod \"852e1076-47cc-4f39-bc85-88719ff826a5\" (UID: \"852e1076-47cc-4f39-bc85-88719ff826a5\") " Nov 01 06:06:34.244096 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:34.244079 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/852e1076-47cc-4f39-bc85-88719ff826a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852e1076-47cc-4f39-bc85-88719ff826a5-maas-api-openshift-ca\") pod \"852e1076-47cc-4f39-bc85-88719ff826a5\" (UID: \"852e1076-47cc-4f39-bc85-88719ff826a5\") " Nov 01 06:06:34.245122 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:34.245098 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/852e1076-47cc-4f39-bc85-88719ff826a5-maas-api-openshift-ca" pod "852e1076-47cc-4f39-bc85-88719ff826a5" (UID: "852e1076-47cc-4f39-bc85-88719ff826a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:06:34.272508 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:34.270087 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/852e1076-47cc-4f39-bc85-88719ff826a5-kube-api-access-zljfg" pod "852e1076-47cc-4f39-bc85-88719ff826a5" (UID: "852e1076-47cc-4f39-bc85-88719ff826a5"). InnerVolumeSpecName "kube-api-access-zljfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:06:34.281694 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:34.281648 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/852e1076-47cc-4f39-bc85-88719ff826a5-maas-api-xks-ca" pod "852e1076-47cc-4f39-bc85-88719ff826a5" (UID: "852e1076-47cc-4f39-bc85-88719ff826a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:06:34.374504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:34.363918 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852e1076-47cc-4f39-bc85-88719ff826a5-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:34.374504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:34.363963 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zljfg\" (UniqueName: \"kubernetes.io/projected/852e1076-47cc-4f39-bc85-88719ff826a5-kube-api-access-zljfg\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:34.374504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:34.363981 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852e1076-47cc-4f39-bc85-88719ff826a5-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:34.399511 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:34.387668 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" podUID="75afe340-0b9c-4b4c-89ab-20d761e6b4f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:06:34.399511 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:34.388683 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" podUID="57cf5e07-36de-48cf-b938-138484cea3a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 01 06:06:35.007444 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:35.007398 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" podUID="adbac784-4ded-4ca3-b985-3e59983fc637" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:06:35.027503 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:35.011365 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" Nov 01 06:06:35.030116 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:35.030089 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c05f3b-r78db" event={"ID":"852e1076-47cc-4f39-bc85-88719ff826a5","Type":"ContainerDied","Data":"dd67e28d1b9d40637982145d46829c5f1fff439267ed77da53a5cf3744274469"} Nov 01 06:06:35.031080 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:35.031059 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd67e28d1b9d40637982145d46829c5f1fff439267ed77da53a5cf3744274469" Nov 01 06:06:35.031219 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:35.029892 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c05f3b-r78db" Nov 01 06:06:35.057330 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:35.057279 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" podUID="21f54836-d1ed-4f87-8f26-7c2d939b2590" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Nov 01 06:06:35.057996 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:35.057969 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" Nov 01 06:06:35.071597 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:35.071557 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" podUID="1878b6fb-f8fe-4f4f-8f4e-056c66fc206a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:06:35.093502 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:35.073068 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" Nov 01 06:06:35.191522 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:35.190496 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" Nov 01 06:06:35.393254 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:35.390868 5040 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" podUID="1878b6fb-f8fe-4f4f-8f4e-056c66fc206a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:06:35.421526 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:35.417796 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" Nov 01 06:06:35.430501 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:35.426922 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" Nov 01 06:06:35.435165 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:35.433643 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" podUID="57cf5e07-36de-48cf-b938-138484cea3a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 06:06:35.435165 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:35.434049 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" Nov 01 06:06:35.582420 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:35.581555 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" Nov 01 06:06:39.972629 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:06:39.972592 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:06:45.700082 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:45.700043 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7585a10d-cx7cs"] Nov 01 06:06:45.700616 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:45.700600 5040 cpu_manager.go:414] "containerMap: removing container" podUID="998ef654-d73e-4a90-9e6a-f45d90cc959c" containerName="revoke-keys" Nov 01 06:06:45.700616 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:45.700617 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="998ef654-d73e-4a90-9e6a-f45d90cc959c" containerName="revoke-keys" Nov 01 06:06:45.700704 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:45.700662 5040 cpu_manager.go:414] "containerMap: removing container" podUID="852e1076-47cc-4f39-bc85-88719ff826a5" containerName="revoke-keys" Nov 01 06:06:45.700704 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:45.700667 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="852e1076-47cc-4f39-bc85-88719ff826a5" containerName="revoke-keys" Nov 01 06:06:45.700779 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:45.700742 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="852e1076-47cc-4f39-bc85-88719ff826a5" containerName="revoke-keys" Nov 01 06:06:45.700779 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:45.700754 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="998ef654-d73e-4a90-9e6a-f45d90cc959c" containerName="revoke-keys" Nov 01 06:06:45.704421 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:45.704404 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7585a10d-cx7cs" Nov 01 06:06:45.761647 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:45.761605 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7585a10d-cx7cs"] Nov 01 06:06:45.807248 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:45.807216 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:06:45.807248 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:45.807216 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:06:45.807248 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:45.807216 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:06:45.807524 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:45.807359 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:06:45.880262 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:45.880225 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx5j8\" (UniqueName: \"kubernetes.io/projected/b900531c-f6b3-4b8f-901d-91e8535a2062-kube-api-access-xx5j8\") pod \"maas-api-revoke-sub-7585a10d-cx7cs\" (UID: \"b900531c-f6b3-4b8f-901d-91e8535a2062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7585a10d-cx7cs" Nov 01 06:06:45.880451 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:45.880307 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b900531c-f6b3-4b8f-901d-91e8535a2062-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7585a10d-cx7cs\" (UID: \"b900531c-f6b3-4b8f-901d-91e8535a2062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7585a10d-cx7cs" Nov 01 06:06:45.880451 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:45.880382 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b900531c-f6b3-4b8f-901d-91e8535a2062-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7585a10d-cx7cs\" (UID: \"b900531c-f6b3-4b8f-901d-91e8535a2062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7585a10d-cx7cs" Nov 01 06:06:45.982082 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:45.981980 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx5j8\" (UniqueName: \"kubernetes.io/projected/b900531c-f6b3-4b8f-901d-91e8535a2062-kube-api-access-xx5j8\") pod \"maas-api-revoke-sub-7585a10d-cx7cs\" (UID: \"b900531c-f6b3-4b8f-901d-91e8535a2062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7585a10d-cx7cs" Nov 01 06:06:45.982082 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:45.982064 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b900531c-f6b3-4b8f-901d-91e8535a2062-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7585a10d-cx7cs\" (UID: \"b900531c-f6b3-4b8f-901d-91e8535a2062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7585a10d-cx7cs" Nov 01 06:06:45.982303 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:45.982114 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b900531c-f6b3-4b8f-901d-91e8535a2062-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7585a10d-cx7cs\" (UID: \"b900531c-f6b3-4b8f-901d-91e8535a2062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7585a10d-cx7cs" Nov 01 06:06:45.982779 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:45.982753 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b900531c-f6b3-4b8f-901d-91e8535a2062-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7585a10d-cx7cs\" (UID: \"b900531c-f6b3-4b8f-901d-91e8535a2062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7585a10d-cx7cs" Nov 01 06:06:45.984664 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:45.984646 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b900531c-f6b3-4b8f-901d-91e8535a2062-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7585a10d-cx7cs\" (UID: \"b900531c-f6b3-4b8f-901d-91e8535a2062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7585a10d-cx7cs" Nov 01 06:06:46.043981 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:46.043944 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx5j8\" (UniqueName: \"kubernetes.io/projected/b900531c-f6b3-4b8f-901d-91e8535a2062-kube-api-access-xx5j8\") pod \"maas-api-revoke-sub-7585a10d-cx7cs\" (UID: \"b900531c-f6b3-4b8f-901d-91e8535a2062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7585a10d-cx7cs" Nov 01 06:06:46.318258 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:46.318217 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7585a10d-cx7cs" Nov 01 06:06:46.694646 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:06:46.694611 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb900531c_f6b3_4b8f_901d_91e8535a2062.slice/crio-fb0496c3c699480ee8d4c556923708580cf75e2a774b422451c997dc7510b8f5 WatchSource:0}: Error finding container fb0496c3c699480ee8d4c556923708580cf75e2a774b422451c997dc7510b8f5: Status 404 returned error can't find the container with id fb0496c3c699480ee8d4c556923708580cf75e2a774b422451c997dc7510b8f5 Nov 01 06:06:46.696358 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:46.696325 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7585a10d-cx7cs"] Nov 01 06:06:47.118513 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:47.118459 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7585a10d-cx7cs" event={"ID":"b900531c-f6b3-4b8f-901d-91e8535a2062","Type":"ContainerStarted","Data":"608b817a645129baa1284fef912293636b095803b1444e89a1e7c468842dc8b0"} Nov 01 06:06:47.118917 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:47.118537 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7585a10d-cx7cs" event={"ID":"b900531c-f6b3-4b8f-901d-91e8535a2062","Type":"ContainerStarted","Data":"fb0496c3c699480ee8d4c556923708580cf75e2a774b422451c997dc7510b8f5"} Nov 01 06:06:47.239237 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:47.239183 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7585a10d-cx7cs" podStartSLOduration=2.239169478 podStartE2EDuration="2.239169478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:06:47.235384009 +0000 UTC m=+1285.890501181" watchObservedRunningTime="2026-11-01 06:06:47.239169478 +0000 UTC m=+1285.894286645" Nov 01 06:06:48.127528 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:48.127471 5040 generic.go:356] "Generic (PLEG): container finished" podID="b900531c-f6b3-4b8f-901d-91e8535a2062" containerID="608b817a645129baa1284fef912293636b095803b1444e89a1e7c468842dc8b0" exitCode=0 Nov 01 06:06:48.127916 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:48.127547 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7585a10d-cx7cs" event={"ID":"b900531c-f6b3-4b8f-901d-91e8535a2062","Type":"ContainerDied","Data":"608b817a645129baa1284fef912293636b095803b1444e89a1e7c468842dc8b0"} Nov 01 06:06:49.298283 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:49.298256 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7585a10d-cx7cs" Nov 01 06:06:49.422858 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:49.422783 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b900531c-f6b3-4b8f-901d-91e8535a2062-kube-api-access-xx5j8\" (UniqueName: \"kubernetes.io/projected/b900531c-f6b3-4b8f-901d-91e8535a2062-kube-api-access-xx5j8\") pod \"b900531c-f6b3-4b8f-901d-91e8535a2062\" (UID: \"b900531c-f6b3-4b8f-901d-91e8535a2062\") " Nov 01 06:06:49.423027 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:49.422878 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b900531c-f6b3-4b8f-901d-91e8535a2062-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b900531c-f6b3-4b8f-901d-91e8535a2062-maas-api-openshift-ca\") pod \"b900531c-f6b3-4b8f-901d-91e8535a2062\" (UID: \"b900531c-f6b3-4b8f-901d-91e8535a2062\") " Nov 01 06:06:49.423027 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:49.422949 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b900531c-f6b3-4b8f-901d-91e8535a2062-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b900531c-f6b3-4b8f-901d-91e8535a2062-maas-api-xks-ca\") pod \"b900531c-f6b3-4b8f-901d-91e8535a2062\" (UID: \"b900531c-f6b3-4b8f-901d-91e8535a2062\") " Nov 01 06:06:49.423253 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:49.423216 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b900531c-f6b3-4b8f-901d-91e8535a2062-maas-api-openshift-ca" pod "b900531c-f6b3-4b8f-901d-91e8535a2062" (UID: "b900531c-f6b3-4b8f-901d-91e8535a2062"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:06:49.425228 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:49.425193 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b900531c-f6b3-4b8f-901d-91e8535a2062-maas-api-xks-ca" pod "b900531c-f6b3-4b8f-901d-91e8535a2062" (UID: "b900531c-f6b3-4b8f-901d-91e8535a2062"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:06:49.425345 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:49.425289 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b900531c-f6b3-4b8f-901d-91e8535a2062-kube-api-access-xx5j8" pod "b900531c-f6b3-4b8f-901d-91e8535a2062" (UID: "b900531c-f6b3-4b8f-901d-91e8535a2062"). InnerVolumeSpecName "kube-api-access-xx5j8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:06:49.524094 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:49.524065 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b900531c-f6b3-4b8f-901d-91e8535a2062-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:49.524211 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:49.524101 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b900531c-f6b3-4b8f-901d-91e8535a2062-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:49.524211 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:49.524117 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx5j8\" (UniqueName: \"kubernetes.io/projected/b900531c-f6b3-4b8f-901d-91e8535a2062-kube-api-access-xx5j8\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:50.146918 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:50.146893 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7585a10d-cx7cs" Nov 01 06:06:50.147088 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:50.146891 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7585a10d-cx7cs" event={"ID":"b900531c-f6b3-4b8f-901d-91e8535a2062","Type":"ContainerDied","Data":"fb0496c3c699480ee8d4c556923708580cf75e2a774b422451c997dc7510b8f5"} Nov 01 06:06:50.147088 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:50.147013 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb0496c3c699480ee8d4c556923708580cf75e2a774b422451c997dc7510b8f5" Nov 01 06:06:50.661682 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:50.661640 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb56174b-g7p4r"] Nov 01 06:06:50.662153 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:50.662140 5040 cpu_manager.go:414] "containerMap: removing container" podUID="b900531c-f6b3-4b8f-901d-91e8535a2062" containerName="revoke-keys" Nov 01 06:06:50.662202 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:50.662153 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b900531c-f6b3-4b8f-901d-91e8535a2062" containerName="revoke-keys" Nov 01 06:06:50.662265 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:50.662256 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="b900531c-f6b3-4b8f-901d-91e8535a2062" containerName="revoke-keys" Nov 01 06:06:50.666035 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:50.666017 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb56174b-g7p4r" Nov 01 06:06:50.723211 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:50.723176 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb56174b-g7p4r"] Nov 01 06:06:50.736903 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:50.736873 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn9wr\" (UniqueName: \"kubernetes.io/projected/b8d4b89a-a81e-4763-a236-5bc2ad562a7e-kube-api-access-cn9wr\") pod \"maas-api-revoke-sub-fb56174b-g7p4r\" (UID: \"b8d4b89a-a81e-4763-a236-5bc2ad562a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb56174b-g7p4r" Nov 01 06:06:50.737071 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:50.736938 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8d4b89a-a81e-4763-a236-5bc2ad562a7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb56174b-g7p4r\" (UID: \"b8d4b89a-a81e-4763-a236-5bc2ad562a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb56174b-g7p4r" Nov 01 06:06:50.737118 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:50.737091 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8d4b89a-a81e-4763-a236-5bc2ad562a7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb56174b-g7p4r\" (UID: \"b8d4b89a-a81e-4763-a236-5bc2ad562a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb56174b-g7p4r" Nov 01 06:06:50.768639 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:50.768603 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:06:50.768639 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:50.768611 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:06:50.769275 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:50.769255 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:06:50.769275 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:50.769267 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:06:50.838579 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:50.838540 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8d4b89a-a81e-4763-a236-5bc2ad562a7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb56174b-g7p4r\" (UID: \"b8d4b89a-a81e-4763-a236-5bc2ad562a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb56174b-g7p4r" Nov 01 06:06:50.838758 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:50.838643 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8d4b89a-a81e-4763-a236-5bc2ad562a7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb56174b-g7p4r\" (UID: \"b8d4b89a-a81e-4763-a236-5bc2ad562a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb56174b-g7p4r" Nov 01 06:06:50.838841 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:50.838809 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn9wr\" (UniqueName: \"kubernetes.io/projected/b8d4b89a-a81e-4763-a236-5bc2ad562a7e-kube-api-access-cn9wr\") pod \"maas-api-revoke-sub-fb56174b-g7p4r\" (UID: \"b8d4b89a-a81e-4763-a236-5bc2ad562a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb56174b-g7p4r" Nov 01 06:06:50.839271 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:50.839246 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8d4b89a-a81e-4763-a236-5bc2ad562a7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb56174b-g7p4r\" (UID: \"b8d4b89a-a81e-4763-a236-5bc2ad562a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb56174b-g7p4r" Nov 01 06:06:50.840946 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:50.840928 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8d4b89a-a81e-4763-a236-5bc2ad562a7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb56174b-g7p4r\" (UID: \"b8d4b89a-a81e-4763-a236-5bc2ad562a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb56174b-g7p4r" Nov 01 06:06:50.900643 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:50.900605 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn9wr\" (UniqueName: \"kubernetes.io/projected/b8d4b89a-a81e-4763-a236-5bc2ad562a7e-kube-api-access-cn9wr\") pod \"maas-api-revoke-sub-fb56174b-g7p4r\" (UID: \"b8d4b89a-a81e-4763-a236-5bc2ad562a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb56174b-g7p4r" Nov 01 06:06:50.979163 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:50.979067 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb56174b-g7p4r" Nov 01 06:06:51.357243 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:06:51.357212 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d4b89a_a81e_4763_a236_5bc2ad562a7e.slice/crio-d8f542f545c78d436eadb11ae2bf56c183feaa93a18fa5803cbdc112d7d942c6 WatchSource:0}: Error finding container d8f542f545c78d436eadb11ae2bf56c183feaa93a18fa5803cbdc112d7d942c6: Status 404 returned error can't find the container with id d8f542f545c78d436eadb11ae2bf56c183feaa93a18fa5803cbdc112d7d942c6 Nov 01 06:06:51.357645 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:51.357619 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb56174b-g7p4r"] Nov 01 06:06:52.165108 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:52.165073 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb56174b-g7p4r" event={"ID":"b8d4b89a-a81e-4763-a236-5bc2ad562a7e","Type":"ContainerStarted","Data":"391befc3245780e53fa36a9f8cdf97744e4943c265c6a350fccc21fdbdc19183"} Nov 01 06:06:52.165526 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:52.165123 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb56174b-g7p4r" event={"ID":"b8d4b89a-a81e-4763-a236-5bc2ad562a7e","Type":"ContainerStarted","Data":"d8f542f545c78d436eadb11ae2bf56c183feaa93a18fa5803cbdc112d7d942c6"} Nov 01 06:06:52.207399 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:52.207365 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-fhpc7-7qkbj"] Nov 01 06:06:52.211584 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:52.211564 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fhpc7-7qkbj" Nov 01 06:06:52.254581 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:52.254546 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f41d65c3-acc5-4de4-a46f-18787ecf7925-archives-path\") pod \"periodic-gathering-fhpc7-7qkbj\" (UID: \"f41d65c3-acc5-4de4-a46f-18787ecf7925\") " pod="openshift-insights/periodic-gathering-fhpc7-7qkbj" Nov 01 06:06:52.254710 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:52.254683 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f41d65c3-acc5-4de4-a46f-18787ecf7925-trusted-ca-bundle\") pod \"periodic-gathering-fhpc7-7qkbj\" (UID: \"f41d65c3-acc5-4de4-a46f-18787ecf7925\") " pod="openshift-insights/periodic-gathering-fhpc7-7qkbj" Nov 01 06:06:52.254874 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:52.254855 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vt7f\" (UniqueName: \"kubernetes.io/projected/f41d65c3-acc5-4de4-a46f-18787ecf7925-kube-api-access-5vt7f\") pod \"periodic-gathering-fhpc7-7qkbj\" (UID: \"f41d65c3-acc5-4de4-a46f-18787ecf7925\") " pod="openshift-insights/periodic-gathering-fhpc7-7qkbj" Nov 01 06:06:52.254941 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:52.254926 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f41d65c3-acc5-4de4-a46f-18787ecf7925-service-ca-bundle\") pod \"periodic-gathering-fhpc7-7qkbj\" (UID: \"f41d65c3-acc5-4de4-a46f-18787ecf7925\") " pod="openshift-insights/periodic-gathering-fhpc7-7qkbj" Nov 01 06:06:52.264565 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:52.264544 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-gvvtc\"" Nov 01 06:06:52.275024 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:52.275001 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-fhpc7-7qkbj"] Nov 01 06:06:52.318270 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:52.318220 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb56174b-g7p4r" podStartSLOduration=2.318207893 podStartE2EDuration="2.318207893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:06:52.3175011 +0000 UTC m=+1290.972618270" watchObservedRunningTime="2026-11-01 06:06:52.318207893 +0000 UTC m=+1290.973325085" Nov 01 06:06:52.356164 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:52.356129 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vt7f\" (UniqueName: \"kubernetes.io/projected/f41d65c3-acc5-4de4-a46f-18787ecf7925-kube-api-access-5vt7f\") pod \"periodic-gathering-fhpc7-7qkbj\" (UID: \"f41d65c3-acc5-4de4-a46f-18787ecf7925\") " pod="openshift-insights/periodic-gathering-fhpc7-7qkbj" Nov 01 06:06:52.356352 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:52.356183 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f41d65c3-acc5-4de4-a46f-18787ecf7925-service-ca-bundle\") pod \"periodic-gathering-fhpc7-7qkbj\" (UID: \"f41d65c3-acc5-4de4-a46f-18787ecf7925\") " pod="openshift-insights/periodic-gathering-fhpc7-7qkbj" Nov 01 06:06:52.356352 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:52.356222 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f41d65c3-acc5-4de4-a46f-18787ecf7925-archives-path\") pod \"periodic-gathering-fhpc7-7qkbj\" (UID: \"f41d65c3-acc5-4de4-a46f-18787ecf7925\") " pod="openshift-insights/periodic-gathering-fhpc7-7qkbj" Nov 01 06:06:52.356352 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:52.356279 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f41d65c3-acc5-4de4-a46f-18787ecf7925-trusted-ca-bundle\") pod \"periodic-gathering-fhpc7-7qkbj\" (UID: \"f41d65c3-acc5-4de4-a46f-18787ecf7925\") " pod="openshift-insights/periodic-gathering-fhpc7-7qkbj" Nov 01 06:06:52.356718 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:52.356695 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f41d65c3-acc5-4de4-a46f-18787ecf7925-archives-path\") pod \"periodic-gathering-fhpc7-7qkbj\" (UID: \"f41d65c3-acc5-4de4-a46f-18787ecf7925\") " pod="openshift-insights/periodic-gathering-fhpc7-7qkbj" Nov 01 06:06:52.356884 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:52.356867 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f41d65c3-acc5-4de4-a46f-18787ecf7925-service-ca-bundle\") pod \"periodic-gathering-fhpc7-7qkbj\" (UID: \"f41d65c3-acc5-4de4-a46f-18787ecf7925\") " pod="openshift-insights/periodic-gathering-fhpc7-7qkbj" Nov 01 06:06:52.357046 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:52.357027 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f41d65c3-acc5-4de4-a46f-18787ecf7925-trusted-ca-bundle\") pod \"periodic-gathering-fhpc7-7qkbj\" (UID: \"f41d65c3-acc5-4de4-a46f-18787ecf7925\") " pod="openshift-insights/periodic-gathering-fhpc7-7qkbj" Nov 01 06:06:52.418464 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:52.418399 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vt7f\" (UniqueName: \"kubernetes.io/projected/f41d65c3-acc5-4de4-a46f-18787ecf7925-kube-api-access-5vt7f\") pod \"periodic-gathering-fhpc7-7qkbj\" (UID: \"f41d65c3-acc5-4de4-a46f-18787ecf7925\") " pod="openshift-insights/periodic-gathering-fhpc7-7qkbj" Nov 01 06:06:52.524691 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:52.524656 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fhpc7-7qkbj" Nov 01 06:06:52.914388 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:52.914361 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-fhpc7-7qkbj"] Nov 01 06:06:52.916889 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:06:52.916859 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf41d65c3_acc5_4de4_a46f_18787ecf7925.slice/crio-3e6efcc7d598345d40a2780ccb402856cdc59f432a8d5687afe48eb632f4e688 WatchSource:0}: Error finding container 3e6efcc7d598345d40a2780ccb402856cdc59f432a8d5687afe48eb632f4e688: Status 404 returned error can't find the container with id 3e6efcc7d598345d40a2780ccb402856cdc59f432a8d5687afe48eb632f4e688 Nov 01 06:06:53.211568 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:53.209870 5040 generic.go:356] "Generic (PLEG): container finished" podID="b8d4b89a-a81e-4763-a236-5bc2ad562a7e" containerID="391befc3245780e53fa36a9f8cdf97744e4943c265c6a350fccc21fdbdc19183" exitCode=0 Nov 01 06:06:53.211568 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:53.210011 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb56174b-g7p4r" event={"ID":"b8d4b89a-a81e-4763-a236-5bc2ad562a7e","Type":"ContainerDied","Data":"391befc3245780e53fa36a9f8cdf97744e4943c265c6a350fccc21fdbdc19183"} Nov 01 06:06:53.218494 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:53.217783 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fhpc7-7qkbj" event={"ID":"f41d65c3-acc5-4de4-a46f-18787ecf7925","Type":"ContainerStarted","Data":"6cb45f503b034a20a145d84cf4ab8255248b65e1a53078fdeae876d3ddbf30ee"} Nov 01 06:06:53.218494 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:53.217850 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fhpc7-7qkbj" event={"ID":"f41d65c3-acc5-4de4-a46f-18787ecf7925","Type":"ContainerStarted","Data":"3e6efcc7d598345d40a2780ccb402856cdc59f432a8d5687afe48eb632f4e688"} Nov 01 06:06:53.444500 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:53.444079 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-fhpc7-7qkbj" podStartSLOduration=1.444058926 podStartE2EDuration="1.444058926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:06:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:06:53.444017902 +0000 UTC m=+1292.099135072" watchObservedRunningTime="2026-11-01 06:06:53.444058926 +0000 UTC m=+1292.099176094" Nov 01 06:06:54.913033 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:54.913000 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb56174b-g7p4r" Nov 01 06:06:55.086530 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:55.086499 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a430364f-z485q"] Nov 01 06:06:55.090542 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:55.090463 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a430364f-z485q"] Nov 01 06:06:55.097829 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:55.097804 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8d4b89a-a81e-4763-a236-5bc2ad562a7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8d4b89a-a81e-4763-a236-5bc2ad562a7e-maas-api-openshift-ca\") pod \"b8d4b89a-a81e-4763-a236-5bc2ad562a7e\" (UID: \"b8d4b89a-a81e-4763-a236-5bc2ad562a7e\") " Nov 01 06:06:55.097937 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:55.097892 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8d4b89a-a81e-4763-a236-5bc2ad562a7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8d4b89a-a81e-4763-a236-5bc2ad562a7e-maas-api-xks-ca\") pod \"b8d4b89a-a81e-4763-a236-5bc2ad562a7e\" (UID: \"b8d4b89a-a81e-4763-a236-5bc2ad562a7e\") " Nov 01 06:06:55.097937 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:55.097928 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8d4b89a-a81e-4763-a236-5bc2ad562a7e-kube-api-access-cn9wr\" (UniqueName: \"kubernetes.io/projected/b8d4b89a-a81e-4763-a236-5bc2ad562a7e-kube-api-access-cn9wr\") pod \"b8d4b89a-a81e-4763-a236-5bc2ad562a7e\" (UID: \"b8d4b89a-a81e-4763-a236-5bc2ad562a7e\") " Nov 01 06:06:55.098242 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:55.098198 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d4b89a-a81e-4763-a236-5bc2ad562a7e-maas-api-openshift-ca" pod "b8d4b89a-a81e-4763-a236-5bc2ad562a7e" (UID: "b8d4b89a-a81e-4763-a236-5bc2ad562a7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:06:55.098611 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:55.098591 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8d4b89a-a81e-4763-a236-5bc2ad562a7e-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:55.100096 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:55.100073 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8d4b89a-a81e-4763-a236-5bc2ad562a7e-maas-api-xks-ca" pod "b8d4b89a-a81e-4763-a236-5bc2ad562a7e" (UID: "b8d4b89a-a81e-4763-a236-5bc2ad562a7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:06:55.100418 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:55.100387 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d4b89a-a81e-4763-a236-5bc2ad562a7e-kube-api-access-cn9wr" pod "b8d4b89a-a81e-4763-a236-5bc2ad562a7e" (UID: "b8d4b89a-a81e-4763-a236-5bc2ad562a7e"). InnerVolumeSpecName "kube-api-access-cn9wr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:06:55.199345 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:55.199268 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8d4b89a-a81e-4763-a236-5bc2ad562a7e-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:55.199345 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:55.199314 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn9wr\" (UniqueName: \"kubernetes.io/projected/b8d4b89a-a81e-4763-a236-5bc2ad562a7e-kube-api-access-cn9wr\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:55.515568 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:55.515536 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7024010b-bb61-4b2b-ad95-fa6ca12062f0" path="/var/lib/kubelet/pods/7024010b-bb61-4b2b-ad95-fa6ca12062f0/volumes" Nov 01 06:06:55.525528 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:55.521887 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb56174b-g7p4r" Nov 01 06:06:55.525528 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:55.521883 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb56174b-g7p4r" event={"ID":"b8d4b89a-a81e-4763-a236-5bc2ad562a7e","Type":"ContainerDied","Data":"d8f542f545c78d436eadb11ae2bf56c183feaa93a18fa5803cbdc112d7d942c6"} Nov 01 06:06:55.525528 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:55.521980 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8f542f545c78d436eadb11ae2bf56c183feaa93a18fa5803cbdc112d7d942c6" Nov 01 06:06:56.448894 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:56.448859 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c50fe9e4-7278t"] Nov 01 06:06:56.449388 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:56.449373 5040 cpu_manager.go:414] "containerMap: removing container" podUID="b8d4b89a-a81e-4763-a236-5bc2ad562a7e" containerName="revoke-keys" Nov 01 06:06:56.449460 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:56.449395 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8d4b89a-a81e-4763-a236-5bc2ad562a7e" containerName="revoke-keys" Nov 01 06:06:56.449572 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:56.449560 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8d4b89a-a81e-4763-a236-5bc2ad562a7e" containerName="revoke-keys" Nov 01 06:06:56.453280 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:56.453265 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50fe9e4-7278t" Nov 01 06:06:56.510547 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:56.510518 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c50fe9e4-7278t"] Nov 01 06:06:56.513289 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:56.513258 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c33506-de91-47b3-9495-fdb20ff7d816-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c50fe9e4-7278t\" (UID: \"06c33506-de91-47b3-9495-fdb20ff7d816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50fe9e4-7278t" Nov 01 06:06:56.513493 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:56.513450 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c33506-de91-47b3-9495-fdb20ff7d816-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c50fe9e4-7278t\" (UID: \"06c33506-de91-47b3-9495-fdb20ff7d816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50fe9e4-7278t" Nov 01 06:06:56.513674 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:56.513648 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2tqg\" (UniqueName: \"kubernetes.io/projected/06c33506-de91-47b3-9495-fdb20ff7d816-kube-api-access-z2tqg\") pod \"maas-api-revoke-sub-c50fe9e4-7278t\" (UID: \"06c33506-de91-47b3-9495-fdb20ff7d816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50fe9e4-7278t" Nov 01 06:06:56.554870 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:56.554844 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:06:56.554870 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:56.554845 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:06:56.555063 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:56.554852 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:06:56.555063 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:56.554920 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:06:56.615020 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:56.614986 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c33506-de91-47b3-9495-fdb20ff7d816-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c50fe9e4-7278t\" (UID: \"06c33506-de91-47b3-9495-fdb20ff7d816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50fe9e4-7278t" Nov 01 06:06:56.615192 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:56.615076 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2tqg\" (UniqueName: \"kubernetes.io/projected/06c33506-de91-47b3-9495-fdb20ff7d816-kube-api-access-z2tqg\") pod \"maas-api-revoke-sub-c50fe9e4-7278t\" (UID: \"06c33506-de91-47b3-9495-fdb20ff7d816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50fe9e4-7278t" Nov 01 06:06:56.615192 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:56.615120 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c33506-de91-47b3-9495-fdb20ff7d816-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c50fe9e4-7278t\" (UID: \"06c33506-de91-47b3-9495-fdb20ff7d816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50fe9e4-7278t" Nov 01 06:06:56.615815 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:56.615790 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c33506-de91-47b3-9495-fdb20ff7d816-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c50fe9e4-7278t\" (UID: \"06c33506-de91-47b3-9495-fdb20ff7d816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50fe9e4-7278t" Nov 01 06:06:56.617356 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:56.617335 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c33506-de91-47b3-9495-fdb20ff7d816-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c50fe9e4-7278t\" (UID: \"06c33506-de91-47b3-9495-fdb20ff7d816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50fe9e4-7278t" Nov 01 06:06:56.676633 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:56.676602 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2tqg\" (UniqueName: \"kubernetes.io/projected/06c33506-de91-47b3-9495-fdb20ff7d816-kube-api-access-z2tqg\") pod \"maas-api-revoke-sub-c50fe9e4-7278t\" (UID: \"06c33506-de91-47b3-9495-fdb20ff7d816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50fe9e4-7278t" Nov 01 06:06:56.766311 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:56.766270 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50fe9e4-7278t" Nov 01 06:06:57.146516 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:06:57.146468 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06c33506_de91_47b3_9495_fdb20ff7d816.slice/crio-fc74974dfb3c4a084e971230de0263e73a6681104aab14b34eb023f45c10a29b WatchSource:0}: Error finding container fc74974dfb3c4a084e971230de0263e73a6681104aab14b34eb023f45c10a29b: Status 404 returned error can't find the container with id fc74974dfb3c4a084e971230de0263e73a6681104aab14b34eb023f45c10a29b Nov 01 06:06:57.148449 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:57.148419 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c50fe9e4-7278t"] Nov 01 06:06:57.539419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:57.539381 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50fe9e4-7278t" event={"ID":"06c33506-de91-47b3-9495-fdb20ff7d816","Type":"ContainerStarted","Data":"f04cc6736afe59ba1350964fe7e8043ab3a2937a6d8f1b368b5d6bddcdb08d39"} Nov 01 06:06:57.539814 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:57.539434 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50fe9e4-7278t" event={"ID":"06c33506-de91-47b3-9495-fdb20ff7d816","Type":"ContainerStarted","Data":"fc74974dfb3c4a084e971230de0263e73a6681104aab14b34eb023f45c10a29b"} Nov 01 06:06:57.658185 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:57.658097 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50fe9e4-7278t" podStartSLOduration=1.658081462 podStartE2EDuration="1.658081462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:06:57.65621114 +0000 UTC m=+1296.311328309" watchObservedRunningTime="2026-11-01 06:06:57.658081462 +0000 UTC m=+1296.313198629" Nov 01 06:06:58.548104 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:58.548069 5040 generic.go:356] "Generic (PLEG): container finished" podID="06c33506-de91-47b3-9495-fdb20ff7d816" containerID="f04cc6736afe59ba1350964fe7e8043ab3a2937a6d8f1b368b5d6bddcdb08d39" exitCode=0 Nov 01 06:06:58.548570 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:58.548153 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50fe9e4-7278t" event={"ID":"06c33506-de91-47b3-9495-fdb20ff7d816","Type":"ContainerDied","Data":"f04cc6736afe59ba1350964fe7e8043ab3a2937a6d8f1b368b5d6bddcdb08d39"} Nov 01 06:06:59.724989 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:59.724964 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50fe9e4-7278t" Nov 01 06:06:59.750063 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:59.750036 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06c33506-de91-47b3-9495-fdb20ff7d816-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c33506-de91-47b3-9495-fdb20ff7d816-maas-api-xks-ca\") pod \"06c33506-de91-47b3-9495-fdb20ff7d816\" (UID: \"06c33506-de91-47b3-9495-fdb20ff7d816\") " Nov 01 06:06:59.750234 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:59.750076 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06c33506-de91-47b3-9495-fdb20ff7d816-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c33506-de91-47b3-9495-fdb20ff7d816-maas-api-openshift-ca\") pod \"06c33506-de91-47b3-9495-fdb20ff7d816\" (UID: \"06c33506-de91-47b3-9495-fdb20ff7d816\") " Nov 01 06:06:59.750302 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:59.750247 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06c33506-de91-47b3-9495-fdb20ff7d816-kube-api-access-z2tqg\" (UniqueName: \"kubernetes.io/projected/06c33506-de91-47b3-9495-fdb20ff7d816-kube-api-access-z2tqg\") pod \"06c33506-de91-47b3-9495-fdb20ff7d816\" (UID: \"06c33506-de91-47b3-9495-fdb20ff7d816\") " Nov 01 06:06:59.750506 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:59.750464 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06c33506-de91-47b3-9495-fdb20ff7d816-maas-api-openshift-ca" pod "06c33506-de91-47b3-9495-fdb20ff7d816" (UID: "06c33506-de91-47b3-9495-fdb20ff7d816"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:06:59.750667 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:59.750646 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c33506-de91-47b3-9495-fdb20ff7d816-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:59.752202 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:59.752176 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06c33506-de91-47b3-9495-fdb20ff7d816-maas-api-xks-ca" pod "06c33506-de91-47b3-9495-fdb20ff7d816" (UID: "06c33506-de91-47b3-9495-fdb20ff7d816"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:06:59.752288 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:59.752240 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06c33506-de91-47b3-9495-fdb20ff7d816-kube-api-access-z2tqg" pod "06c33506-de91-47b3-9495-fdb20ff7d816" (UID: "06c33506-de91-47b3-9495-fdb20ff7d816"). InnerVolumeSpecName "kube-api-access-z2tqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:06:59.851492 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:59.851400 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2tqg\" (UniqueName: \"kubernetes.io/projected/06c33506-de91-47b3-9495-fdb20ff7d816-kube-api-access-z2tqg\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:59.851492 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:06:59.851432 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c33506-de91-47b3-9495-fdb20ff7d816-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:07:00.565011 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:00.564972 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50fe9e4-7278t" Nov 01 06:07:00.565011 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:00.565003 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c50fe9e4-7278t" event={"ID":"06c33506-de91-47b3-9495-fdb20ff7d816","Type":"ContainerDied","Data":"fc74974dfb3c4a084e971230de0263e73a6681104aab14b34eb023f45c10a29b"} Nov 01 06:07:00.565331 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:00.565054 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc74974dfb3c4a084e971230de0263e73a6681104aab14b34eb023f45c10a29b" Nov 01 06:07:09.383603 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:09.383554 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" podUID="75afe340-0b9c-4b4c-89ab-20d761e6b4f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 06:07:09.587787 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:09.587728 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" podUID="57cf5e07-36de-48cf-b938-138484cea3a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:07:10.161090 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:07:10.161048 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:07:10.561131 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:10.561091 5040 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 06:07:10.561581 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:10.561170 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" podUID="bbafbb5e-f923-447d-a4ac-8a0d4ef73752" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 06:07:12.193430 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:12.193398 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc1c06e9-t8h55"] Nov 01 06:07:12.193928 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:12.193919 5040 cpu_manager.go:414] "containerMap: removing container" podUID="06c33506-de91-47b3-9495-fdb20ff7d816" containerName="revoke-keys" Nov 01 06:07:12.193983 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:12.193931 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06c33506-de91-47b3-9495-fdb20ff7d816" containerName="revoke-keys" Nov 01 06:07:12.194036 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:12.194020 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="06c33506-de91-47b3-9495-fdb20ff7d816" containerName="revoke-keys" Nov 01 06:07:12.197247 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:12.197230 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1c06e9-t8h55" Nov 01 06:07:12.269185 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:12.269150 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc1c06e9-t8h55"] Nov 01 06:07:12.299853 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:12.299803 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:07:12.299853 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:12.299802 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:07:12.299853 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:12.299811 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:07:12.300405 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:12.300385 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:07:12.334987 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:12.334957 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0de30789-420b-4616-a383-b4d9a571e0af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc1c06e9-t8h55\" (UID: \"0de30789-420b-4616-a383-b4d9a571e0af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1c06e9-t8h55" Nov 01 06:07:12.335149 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:12.335013 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dfqc\" (UniqueName: \"kubernetes.io/projected/0de30789-420b-4616-a383-b4d9a571e0af-kube-api-access-6dfqc\") pod \"maas-api-revoke-sub-fc1c06e9-t8h55\" (UID: \"0de30789-420b-4616-a383-b4d9a571e0af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1c06e9-t8h55" Nov 01 06:07:12.335149 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:12.335112 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0de30789-420b-4616-a383-b4d9a571e0af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc1c06e9-t8h55\" (UID: \"0de30789-420b-4616-a383-b4d9a571e0af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1c06e9-t8h55" Nov 01 06:07:12.436103 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:12.436072 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0de30789-420b-4616-a383-b4d9a571e0af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc1c06e9-t8h55\" (UID: \"0de30789-420b-4616-a383-b4d9a571e0af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1c06e9-t8h55" Nov 01 06:07:12.436319 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:12.436202 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0de30789-420b-4616-a383-b4d9a571e0af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc1c06e9-t8h55\" (UID: \"0de30789-420b-4616-a383-b4d9a571e0af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1c06e9-t8h55" Nov 01 06:07:12.436319 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:12.436238 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dfqc\" (UniqueName: \"kubernetes.io/projected/0de30789-420b-4616-a383-b4d9a571e0af-kube-api-access-6dfqc\") pod \"maas-api-revoke-sub-fc1c06e9-t8h55\" (UID: \"0de30789-420b-4616-a383-b4d9a571e0af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1c06e9-t8h55" Nov 01 06:07:12.437059 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:12.437035 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0de30789-420b-4616-a383-b4d9a571e0af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc1c06e9-t8h55\" (UID: \"0de30789-420b-4616-a383-b4d9a571e0af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1c06e9-t8h55" Nov 01 06:07:12.438526 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:12.438471 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0de30789-420b-4616-a383-b4d9a571e0af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc1c06e9-t8h55\" (UID: \"0de30789-420b-4616-a383-b4d9a571e0af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1c06e9-t8h55" Nov 01 06:07:12.497852 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:12.497817 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dfqc\" (UniqueName: \"kubernetes.io/projected/0de30789-420b-4616-a383-b4d9a571e0af-kube-api-access-6dfqc\") pod \"maas-api-revoke-sub-fc1c06e9-t8h55\" (UID: \"0de30789-420b-4616-a383-b4d9a571e0af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1c06e9-t8h55" Nov 01 06:07:12.511066 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:12.511040 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1c06e9-t8h55" Nov 01 06:07:12.887504 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:07:12.887445 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de30789_420b_4616_a383_b4d9a571e0af.slice/crio-31946530c58bb3289b332547b755f58778701d4ed1d21a5da114d4458ac93588 WatchSource:0}: Error finding container 31946530c58bb3289b332547b755f58778701d4ed1d21a5da114d4458ac93588: Status 404 returned error can't find the container with id 31946530c58bb3289b332547b755f58778701d4ed1d21a5da114d4458ac93588 Nov 01 06:07:12.888119 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:12.888092 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc1c06e9-t8h55"] Nov 01 06:07:13.757813 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:13.757766 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1c06e9-t8h55" event={"ID":"0de30789-420b-4616-a383-b4d9a571e0af","Type":"ContainerStarted","Data":"059ceceeda90b0b4496b1a298885e682cfd9a64c1e8fce0611e11eba1c7fff31"} Nov 01 06:07:13.758266 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:13.757834 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1c06e9-t8h55" event={"ID":"0de30789-420b-4616-a383-b4d9a571e0af","Type":"ContainerStarted","Data":"31946530c58bb3289b332547b755f58778701d4ed1d21a5da114d4458ac93588"} Nov 01 06:07:13.880874 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:13.880811 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1c06e9-t8h55" podStartSLOduration=1.880796929 podStartE2EDuration="1.880796929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:07:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:07:13.876865008 +0000 UTC m=+1312.531982173" watchObservedRunningTime="2026-11-01 06:07:13.880796929 +0000 UTC m=+1312.535914120" Nov 01 06:07:14.766937 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:14.766899 5040 generic.go:356] "Generic (PLEG): container finished" podID="0de30789-420b-4616-a383-b4d9a571e0af" containerID="059ceceeda90b0b4496b1a298885e682cfd9a64c1e8fce0611e11eba1c7fff31" exitCode=0 Nov 01 06:07:14.766937 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:14.766940 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1c06e9-t8h55" event={"ID":"0de30789-420b-4616-a383-b4d9a571e0af","Type":"ContainerDied","Data":"059ceceeda90b0b4496b1a298885e682cfd9a64c1e8fce0611e11eba1c7fff31"} Nov 01 06:07:15.931586 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:15.931548 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1c06e9-t8h55" Nov 01 06:07:16.079096 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:16.079008 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0de30789-420b-4616-a383-b4d9a571e0af-kube-api-access-6dfqc\" (UniqueName: \"kubernetes.io/projected/0de30789-420b-4616-a383-b4d9a571e0af-kube-api-access-6dfqc\") pod \"0de30789-420b-4616-a383-b4d9a571e0af\" (UID: \"0de30789-420b-4616-a383-b4d9a571e0af\") " Nov 01 06:07:16.079096 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:16.079065 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0de30789-420b-4616-a383-b4d9a571e0af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0de30789-420b-4616-a383-b4d9a571e0af-maas-api-xks-ca\") pod \"0de30789-420b-4616-a383-b4d9a571e0af\" (UID: \"0de30789-420b-4616-a383-b4d9a571e0af\") " Nov 01 06:07:16.079333 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:16.079149 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0de30789-420b-4616-a383-b4d9a571e0af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0de30789-420b-4616-a383-b4d9a571e0af-maas-api-openshift-ca\") pod \"0de30789-420b-4616-a383-b4d9a571e0af\" (UID: \"0de30789-420b-4616-a383-b4d9a571e0af\") " Nov 01 06:07:16.079631 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:16.079595 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0de30789-420b-4616-a383-b4d9a571e0af-maas-api-openshift-ca" pod "0de30789-420b-4616-a383-b4d9a571e0af" (UID: "0de30789-420b-4616-a383-b4d9a571e0af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:07:16.081254 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:16.081225 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0de30789-420b-4616-a383-b4d9a571e0af-kube-api-access-6dfqc" pod "0de30789-420b-4616-a383-b4d9a571e0af" (UID: "0de30789-420b-4616-a383-b4d9a571e0af"). InnerVolumeSpecName "kube-api-access-6dfqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:07:16.081354 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:16.081279 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0de30789-420b-4616-a383-b4d9a571e0af-maas-api-xks-ca" pod "0de30789-420b-4616-a383-b4d9a571e0af" (UID: "0de30789-420b-4616-a383-b4d9a571e0af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:07:16.180573 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:16.180528 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dfqc\" (UniqueName: \"kubernetes.io/projected/0de30789-420b-4616-a383-b4d9a571e0af-kube-api-access-6dfqc\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:07:16.180573 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:16.180566 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0de30789-420b-4616-a383-b4d9a571e0af-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:07:16.180573 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:16.180581 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0de30789-420b-4616-a383-b4d9a571e0af-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:07:16.783793 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:16.783765 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1c06e9-t8h55" Nov 01 06:07:16.783977 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:16.783803 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1c06e9-t8h55" event={"ID":"0de30789-420b-4616-a383-b4d9a571e0af","Type":"ContainerDied","Data":"31946530c58bb3289b332547b755f58778701d4ed1d21a5da114d4458ac93588"} Nov 01 06:07:16.783977 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:16.783849 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31946530c58bb3289b332547b755f58778701d4ed1d21a5da114d4458ac93588" Nov 01 06:07:21.051390 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:21.051355 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c08529d-wsqpq"] Nov 01 06:07:21.103139 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:21.103101 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c08529d-wsqpq"] Nov 01 06:07:21.561449 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:21.561372 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="948a4c8b-5e28-4fe0-8933-4f0ac460002e" path="/var/lib/kubelet/pods/948a4c8b-5e28-4fe0-8933-4f0ac460002e/volumes" Nov 01 06:07:22.842119 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:22.842086 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-7d966b6c54-v7flb"] Nov 01 06:07:22.842678 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:22.842661 5040 cpu_manager.go:414] "containerMap: removing container" podUID="0de30789-420b-4616-a383-b4d9a571e0af" containerName="revoke-keys" Nov 01 06:07:22.842742 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:22.842679 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0de30789-420b-4616-a383-b4d9a571e0af" containerName="revoke-keys" Nov 01 06:07:22.842792 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:22.842782 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="0de30789-420b-4616-a383-b4d9a571e0af" containerName="revoke-keys" Nov 01 06:07:22.848201 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:22.848180 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-7d966b6c54-v7flb" Nov 01 06:07:22.901469 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:22.901431 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2d3f8d\"" Nov 01 06:07:22.904562 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:22.904532 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-7d966b6c54-v7flb"] Nov 01 06:07:22.972867 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:22.972830 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2d3f8d-dockercfg-pxf4b\"" Nov 01 06:07:23.001213 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:23.001180 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07af255e-cc07-4e90-bdba-828ac197fa53-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2d3f8d-7d966b6c54-v7flb\" (UID: \"07af255e-cc07-4e90-bdba-828ac197fa53\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-7d966b6c54-v7flb" Nov 01 06:07:23.001371 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:23.001232 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9ltd\" (UniqueName: \"kubernetes.io/projected/07af255e-cc07-4e90-bdba-828ac197fa53-kube-api-access-p9ltd\") pod \"payload-processing-e2e-ipp-b-2d3f8d-7d966b6c54-v7flb\" (UID: \"07af255e-cc07-4e90-bdba-828ac197fa53\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-7d966b6c54-v7flb" Nov 01 06:07:23.061299 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:23.061264 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7da6a9db-zgk6m"] Nov 01 06:07:23.065615 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:23.065589 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da6a9db-zgk6m" Nov 01 06:07:23.102740 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:23.102645 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07af255e-cc07-4e90-bdba-828ac197fa53-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2d3f8d-7d966b6c54-v7flb\" (UID: \"07af255e-cc07-4e90-bdba-828ac197fa53\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-7d966b6c54-v7flb" Nov 01 06:07:23.102740 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:23.102711 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9ltd\" (UniqueName: \"kubernetes.io/projected/07af255e-cc07-4e90-bdba-828ac197fa53-kube-api-access-p9ltd\") pod \"payload-processing-e2e-ipp-b-2d3f8d-7d966b6c54-v7flb\" (UID: \"07af255e-cc07-4e90-bdba-828ac197fa53\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-7d966b6c54-v7flb" Nov 01 06:07:23.103446 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:23.103422 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07af255e-cc07-4e90-bdba-828ac197fa53-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2d3f8d-7d966b6c54-v7flb\" (UID: \"07af255e-cc07-4e90-bdba-828ac197fa53\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-7d966b6c54-v7flb" Nov 01 06:07:23.122555 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:23.120955 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:07:23.140411 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:23.140378 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:07:23.140644 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:23.140435 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:07:23.140644 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:23.140584 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:07:23.143867 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:23.143834 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7da6a9db-zgk6m"] Nov 01 06:07:23.164675 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:23.164638 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9ltd\" (UniqueName: \"kubernetes.io/projected/07af255e-cc07-4e90-bdba-828ac197fa53-kube-api-access-p9ltd\") pod \"payload-processing-e2e-ipp-b-2d3f8d-7d966b6c54-v7flb\" (UID: \"07af255e-cc07-4e90-bdba-828ac197fa53\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-7d966b6c54-v7flb" Nov 01 06:07:23.170582 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:23.170543 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-7d966b6c54-v7flb" Nov 01 06:07:23.227507 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:23.217997 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05a3c38e-d470-4e9b-b444-20730fa25f16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7da6a9db-zgk6m\" (UID: \"05a3c38e-d470-4e9b-b444-20730fa25f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da6a9db-zgk6m" Nov 01 06:07:23.227507 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:23.218056 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05a3c38e-d470-4e9b-b444-20730fa25f16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7da6a9db-zgk6m\" (UID: \"05a3c38e-d470-4e9b-b444-20730fa25f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da6a9db-zgk6m" Nov 01 06:07:23.227507 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:23.218096 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtkb9\" (UniqueName: \"kubernetes.io/projected/05a3c38e-d470-4e9b-b444-20730fa25f16-kube-api-access-qtkb9\") pod \"maas-api-revoke-sub-7da6a9db-zgk6m\" (UID: \"05a3c38e-d470-4e9b-b444-20730fa25f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da6a9db-zgk6m" Nov 01 06:07:23.336769 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:23.336733 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05a3c38e-d470-4e9b-b444-20730fa25f16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7da6a9db-zgk6m\" (UID: \"05a3c38e-d470-4e9b-b444-20730fa25f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da6a9db-zgk6m" Nov 01 06:07:23.336934 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:23.336794 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05a3c38e-d470-4e9b-b444-20730fa25f16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7da6a9db-zgk6m\" (UID: \"05a3c38e-d470-4e9b-b444-20730fa25f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da6a9db-zgk6m" Nov 01 06:07:23.336934 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:23.336845 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtkb9\" (UniqueName: \"kubernetes.io/projected/05a3c38e-d470-4e9b-b444-20730fa25f16-kube-api-access-qtkb9\") pod \"maas-api-revoke-sub-7da6a9db-zgk6m\" (UID: \"05a3c38e-d470-4e9b-b444-20730fa25f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da6a9db-zgk6m" Nov 01 06:07:23.337656 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:23.337625 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05a3c38e-d470-4e9b-b444-20730fa25f16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7da6a9db-zgk6m\" (UID: \"05a3c38e-d470-4e9b-b444-20730fa25f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da6a9db-zgk6m" Nov 01 06:07:23.341170 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:23.341121 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05a3c38e-d470-4e9b-b444-20730fa25f16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7da6a9db-zgk6m\" (UID: \"05a3c38e-d470-4e9b-b444-20730fa25f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da6a9db-zgk6m" Nov 01 06:07:23.411547 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:23.402985 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtkb9\" (UniqueName: \"kubernetes.io/projected/05a3c38e-d470-4e9b-b444-20730fa25f16-kube-api-access-qtkb9\") pod \"maas-api-revoke-sub-7da6a9db-zgk6m\" (UID: \"05a3c38e-d470-4e9b-b444-20730fa25f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da6a9db-zgk6m" Nov 01 06:07:23.600813 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:23.600745 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-7d966b6c54-v7flb"] Nov 01 06:07:23.606662 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:07:23.606621 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07af255e_cc07_4e90_bdba_828ac197fa53.slice/crio-dd89aedf4d9bd3f41b1056cef28e1935a6ce366c421cb935f981ac2389e1a194 WatchSource:0}: Error finding container dd89aedf4d9bd3f41b1056cef28e1935a6ce366c421cb935f981ac2389e1a194: Status 404 returned error can't find the container with id dd89aedf4d9bd3f41b1056cef28e1935a6ce366c421cb935f981ac2389e1a194 Nov 01 06:07:23.683722 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:23.683687 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da6a9db-zgk6m" Nov 01 06:07:23.856692 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:23.856651 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-7d966b6c54-v7flb" event={"ID":"07af255e-cc07-4e90-bdba-828ac197fa53","Type":"ContainerStarted","Data":"6aa26c5f51e3c152595122f9649d66fb3763dc5a6554f0a5d62d5f6966206f7e"} Nov 01 06:07:23.857079 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:23.856724 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-7d966b6c54-v7flb" Nov 01 06:07:23.857079 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:23.856748 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-7d966b6c54-v7flb" event={"ID":"07af255e-cc07-4e90-bdba-828ac197fa53","Type":"ContainerStarted","Data":"dd89aedf4d9bd3f41b1056cef28e1935a6ce366c421cb935f981ac2389e1a194"} Nov 01 06:07:23.982343 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:23.982218 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-7d966b6c54-v7flb" podStartSLOduration=1.9821982550000001 podStartE2EDuration="1.982198255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:07:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:07:23.980891513 +0000 UTC m=+1322.636008682" watchObservedRunningTime="2026-11-01 06:07:23.982198255 +0000 UTC m=+1322.637315429" Nov 01 06:07:24.095055 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:07:24.095022 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05a3c38e_d470_4e9b_b444_20730fa25f16.slice/crio-a2cac64e964cd7be86dde073a8062d013ba5e3ac9f75708aa6cd58ad4899e68e WatchSource:0}: Error finding container a2cac64e964cd7be86dde073a8062d013ba5e3ac9f75708aa6cd58ad4899e68e: Status 404 returned error can't find the container with id a2cac64e964cd7be86dde073a8062d013ba5e3ac9f75708aa6cd58ad4899e68e Nov 01 06:07:24.095836 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:24.095805 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7da6a9db-zgk6m"] Nov 01 06:07:24.881571 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:24.881526 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da6a9db-zgk6m" event={"ID":"05a3c38e-d470-4e9b-b444-20730fa25f16","Type":"ContainerStarted","Data":"15aa1987af6e52e5fc959721bcde3aba2bae8d7f250f69603921956dbde14b2d"} Nov 01 06:07:24.881928 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:24.881631 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da6a9db-zgk6m" event={"ID":"05a3c38e-d470-4e9b-b444-20730fa25f16","Type":"ContainerStarted","Data":"a2cac64e964cd7be86dde073a8062d013ba5e3ac9f75708aa6cd58ad4899e68e"} Nov 01 06:07:25.032664 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:25.032602 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da6a9db-zgk6m" podStartSLOduration=2.032586382 podStartE2EDuration="2.032586382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:07:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:07:25.012209827 +0000 UTC m=+1323.667342036" watchObservedRunningTime="2026-11-01 06:07:25.032586382 +0000 UTC m=+1323.687703620" Nov 01 06:07:25.891784 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:25.891747 5040 generic.go:356] "Generic (PLEG): container finished" podID="05a3c38e-d470-4e9b-b444-20730fa25f16" containerID="15aa1987af6e52e5fc959721bcde3aba2bae8d7f250f69603921956dbde14b2d" exitCode=0 Nov 01 06:07:25.892190 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:25.891808 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da6a9db-zgk6m" event={"ID":"05a3c38e-d470-4e9b-b444-20730fa25f16","Type":"ContainerDied","Data":"15aa1987af6e52e5fc959721bcde3aba2bae8d7f250f69603921956dbde14b2d"} Nov 01 06:07:28.166015 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:28.165981 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da6a9db-zgk6m" Nov 01 06:07:28.258613 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:28.258565 5040 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" podUID="57cf5e07-36de-48cf-b938-138484cea3a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:07:28.258954 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:28.258592 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" podUID="21f54836-d1ed-4f87-8f26-7c2d939b2590" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:07:28.263873 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:28.263843 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05a3c38e-d470-4e9b-b444-20730fa25f16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05a3c38e-d470-4e9b-b444-20730fa25f16-maas-api-openshift-ca\") pod \"05a3c38e-d470-4e9b-b444-20730fa25f16\" (UID: \"05a3c38e-d470-4e9b-b444-20730fa25f16\") " Nov 01 06:07:28.264630 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:28.264606 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05a3c38e-d470-4e9b-b444-20730fa25f16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05a3c38e-d470-4e9b-b444-20730fa25f16-maas-api-xks-ca\") pod \"05a3c38e-d470-4e9b-b444-20730fa25f16\" (UID: \"05a3c38e-d470-4e9b-b444-20730fa25f16\") " Nov 01 06:07:28.264855 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:28.264835 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05a3c38e-d470-4e9b-b444-20730fa25f16-kube-api-access-qtkb9\" (UniqueName: \"kubernetes.io/projected/05a3c38e-d470-4e9b-b444-20730fa25f16-kube-api-access-qtkb9\") pod \"05a3c38e-d470-4e9b-b444-20730fa25f16\" (UID: \"05a3c38e-d470-4e9b-b444-20730fa25f16\") " Nov 01 06:07:28.265695 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:28.264384 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05a3c38e-d470-4e9b-b444-20730fa25f16-maas-api-openshift-ca" pod "05a3c38e-d470-4e9b-b444-20730fa25f16" (UID: "05a3c38e-d470-4e9b-b444-20730fa25f16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:07:28.280053 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:28.280020 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a3c38e-d470-4e9b-b444-20730fa25f16-kube-api-access-qtkb9" pod "05a3c38e-d470-4e9b-b444-20730fa25f16" (UID: "05a3c38e-d470-4e9b-b444-20730fa25f16"). InnerVolumeSpecName "kube-api-access-qtkb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:07:28.285062 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:28.285032 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05a3c38e-d470-4e9b-b444-20730fa25f16-maas-api-xks-ca" pod "05a3c38e-d470-4e9b-b444-20730fa25f16" (UID: "05a3c38e-d470-4e9b-b444-20730fa25f16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:07:28.368513 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:28.367591 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05a3c38e-d470-4e9b-b444-20730fa25f16-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:07:28.368513 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:28.367627 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05a3c38e-d470-4e9b-b444-20730fa25f16-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:07:28.368513 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:28.367645 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtkb9\" (UniqueName: \"kubernetes.io/projected/05a3c38e-d470-4e9b-b444-20730fa25f16-kube-api-access-qtkb9\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:07:28.406608 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:28.406559 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" podUID="75afe340-0b9c-4b4c-89ab-20d761e6b4f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:07:28.433663 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:28.433614 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" podUID="adbac784-4ded-4ca3-b985-3e59983fc637" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:07:28.447376 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:28.433910 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" podUID="1878b6fb-f8fe-4f4f-8f4e-056c66fc206a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Nov 01 06:07:28.615691 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:28.615583 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" podUID="57cf5e07-36de-48cf-b938-138484cea3a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:07:29.060628 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:29.060579 5040 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 06:07:29.060836 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:29.060654 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" podUID="313135e2-fa8e-467d-ba47-91b11869b8f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 06:07:29.215564 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:29.215524 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da6a9db-zgk6m" event={"ID":"05a3c38e-d470-4e9b-b444-20730fa25f16","Type":"ContainerDied","Data":"a2cac64e964cd7be86dde073a8062d013ba5e3ac9f75708aa6cd58ad4899e68e"} Nov 01 06:07:29.216151 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:29.216130 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2cac64e964cd7be86dde073a8062d013ba5e3ac9f75708aa6cd58ad4899e68e" Nov 01 06:07:29.216331 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:29.216316 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da6a9db-zgk6m" Nov 01 06:07:29.265569 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:29.263563 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" podUID="21f54836-d1ed-4f87-8f26-7c2d939b2590" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Nov 01 06:07:29.265569 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:29.264351 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" Nov 01 06:07:29.379576 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:29.378995 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" Nov 01 06:07:29.449620 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:29.449579 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" podUID="adbac784-4ded-4ca3-b985-3e59983fc637" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 01 06:07:29.450231 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:29.450211 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" Nov 01 06:07:29.499207 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:29.499179 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" Nov 01 06:07:31.503332 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:31.503300 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cde7b767-jkxfz"] Nov 01 06:07:31.503867 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:31.503852 5040 cpu_manager.go:414] "containerMap: removing container" podUID="05a3c38e-d470-4e9b-b444-20730fa25f16" containerName="revoke-keys" Nov 01 06:07:31.503914 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:31.503869 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05a3c38e-d470-4e9b-b444-20730fa25f16" containerName="revoke-keys" Nov 01 06:07:31.503986 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:31.503975 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="05a3c38e-d470-4e9b-b444-20730fa25f16" containerName="revoke-keys" Nov 01 06:07:31.513107 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:31.513084 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde7b767-jkxfz" Nov 01 06:07:31.566022 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:31.565984 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cde7b767-jkxfz"] Nov 01 06:07:31.608091 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:31.608044 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdkg7\" (UniqueName: \"kubernetes.io/projected/64837f7d-6b63-40de-a034-35701ceda44f-kube-api-access-mdkg7\") pod \"maas-api-revoke-sub-cde7b767-jkxfz\" (UID: \"64837f7d-6b63-40de-a034-35701ceda44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde7b767-jkxfz" Nov 01 06:07:31.608293 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:31.608274 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64837f7d-6b63-40de-a034-35701ceda44f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cde7b767-jkxfz\" (UID: \"64837f7d-6b63-40de-a034-35701ceda44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde7b767-jkxfz" Nov 01 06:07:31.608364 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:31.608348 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64837f7d-6b63-40de-a034-35701ceda44f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cde7b767-jkxfz\" (UID: \"64837f7d-6b63-40de-a034-35701ceda44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde7b767-jkxfz" Nov 01 06:07:31.610309 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:31.610280 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:07:31.610309 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:31.610288 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:07:31.610309 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:31.610307 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:07:31.615975 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:31.615959 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:07:31.710038 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:31.709999 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64837f7d-6b63-40de-a034-35701ceda44f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cde7b767-jkxfz\" (UID: \"64837f7d-6b63-40de-a034-35701ceda44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde7b767-jkxfz" Nov 01 06:07:31.710258 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:31.710070 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64837f7d-6b63-40de-a034-35701ceda44f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cde7b767-jkxfz\" (UID: \"64837f7d-6b63-40de-a034-35701ceda44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde7b767-jkxfz" Nov 01 06:07:31.710258 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:31.710151 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdkg7\" (UniqueName: \"kubernetes.io/projected/64837f7d-6b63-40de-a034-35701ceda44f-kube-api-access-mdkg7\") pod \"maas-api-revoke-sub-cde7b767-jkxfz\" (UID: \"64837f7d-6b63-40de-a034-35701ceda44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde7b767-jkxfz" Nov 01 06:07:31.710836 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:31.710808 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64837f7d-6b63-40de-a034-35701ceda44f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cde7b767-jkxfz\" (UID: \"64837f7d-6b63-40de-a034-35701ceda44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde7b767-jkxfz" Nov 01 06:07:31.712571 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:31.712551 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64837f7d-6b63-40de-a034-35701ceda44f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cde7b767-jkxfz\" (UID: \"64837f7d-6b63-40de-a034-35701ceda44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde7b767-jkxfz" Nov 01 06:07:31.772030 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:31.771960 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdkg7\" (UniqueName: \"kubernetes.io/projected/64837f7d-6b63-40de-a034-35701ceda44f-kube-api-access-mdkg7\") pod \"maas-api-revoke-sub-cde7b767-jkxfz\" (UID: \"64837f7d-6b63-40de-a034-35701ceda44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde7b767-jkxfz" Nov 01 06:07:31.827094 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:31.827051 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde7b767-jkxfz" Nov 01 06:07:32.206022 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:07:32.205992 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64837f7d_6b63_40de_a034_35701ceda44f.slice/crio-9b3448ad3ca868049ef156df29d44c44be41577c61a307ef0e3b3acfafeb5c16 WatchSource:0}: Error finding container 9b3448ad3ca868049ef156df29d44c44be41577c61a307ef0e3b3acfafeb5c16: Status 404 returned error can't find the container with id 9b3448ad3ca868049ef156df29d44c44be41577c61a307ef0e3b3acfafeb5c16 Nov 01 06:07:32.208042 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:32.208013 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cde7b767-jkxfz"] Nov 01 06:07:32.238554 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:32.238514 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde7b767-jkxfz" event={"ID":"64837f7d-6b63-40de-a034-35701ceda44f","Type":"ContainerStarted","Data":"9b3448ad3ca868049ef156df29d44c44be41577c61a307ef0e3b3acfafeb5c16"} Nov 01 06:07:33.248862 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:33.248823 5040 generic.go:356] "Generic (PLEG): container finished" podID="64837f7d-6b63-40de-a034-35701ceda44f" containerID="83b8efb13e934fb4ec2e601a27f77072bec403e586ccca908018025d73d442ba" exitCode=0 Nov 01 06:07:33.250856 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:33.248913 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde7b767-jkxfz" event={"ID":"64837f7d-6b63-40de-a034-35701ceda44f","Type":"ContainerDied","Data":"83b8efb13e934fb4ec2e601a27f77072bec403e586ccca908018025d73d442ba"} Nov 01 06:07:34.433222 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:34.433193 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde7b767-jkxfz" Nov 01 06:07:34.543403 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:34.543308 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64837f7d-6b63-40de-a034-35701ceda44f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64837f7d-6b63-40de-a034-35701ceda44f-maas-api-openshift-ca\") pod \"64837f7d-6b63-40de-a034-35701ceda44f\" (UID: \"64837f7d-6b63-40de-a034-35701ceda44f\") " Nov 01 06:07:34.543403 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:34.543369 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64837f7d-6b63-40de-a034-35701ceda44f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64837f7d-6b63-40de-a034-35701ceda44f-maas-api-xks-ca\") pod \"64837f7d-6b63-40de-a034-35701ceda44f\" (UID: \"64837f7d-6b63-40de-a034-35701ceda44f\") " Nov 01 06:07:34.543680 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:34.543414 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64837f7d-6b63-40de-a034-35701ceda44f-kube-api-access-mdkg7\" (UniqueName: \"kubernetes.io/projected/64837f7d-6b63-40de-a034-35701ceda44f-kube-api-access-mdkg7\") pod \"64837f7d-6b63-40de-a034-35701ceda44f\" (UID: \"64837f7d-6b63-40de-a034-35701ceda44f\") " Nov 01 06:07:34.543924 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:34.543894 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64837f7d-6b63-40de-a034-35701ceda44f-maas-api-openshift-ca" pod "64837f7d-6b63-40de-a034-35701ceda44f" (UID: "64837f7d-6b63-40de-a034-35701ceda44f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:07:34.545541 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:34.545512 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64837f7d-6b63-40de-a034-35701ceda44f-maas-api-xks-ca" pod "64837f7d-6b63-40de-a034-35701ceda44f" (UID: "64837f7d-6b63-40de-a034-35701ceda44f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:07:34.545651 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:34.545605 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64837f7d-6b63-40de-a034-35701ceda44f-kube-api-access-mdkg7" pod "64837f7d-6b63-40de-a034-35701ceda44f" (UID: "64837f7d-6b63-40de-a034-35701ceda44f"). InnerVolumeSpecName "kube-api-access-mdkg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:07:34.645267 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:34.645233 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64837f7d-6b63-40de-a034-35701ceda44f-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:07:34.645267 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:34.645266 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64837f7d-6b63-40de-a034-35701ceda44f-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:07:34.645516 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:34.645278 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdkg7\" (UniqueName: \"kubernetes.io/projected/64837f7d-6b63-40de-a034-35701ceda44f-kube-api-access-mdkg7\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:07:34.884404 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:34.884324 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-7d966b6c54-v7flb" Nov 01 06:07:35.268541 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:35.268502 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde7b767-jkxfz" Nov 01 06:07:35.268541 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:35.268512 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde7b767-jkxfz" event={"ID":"64837f7d-6b63-40de-a034-35701ceda44f","Type":"ContainerDied","Data":"9b3448ad3ca868049ef156df29d44c44be41577c61a307ef0e3b3acfafeb5c16"} Nov 01 06:07:35.268795 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:35.268568 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b3448ad3ca868049ef156df29d44c44be41577c61a307ef0e3b3acfafeb5c16" Nov 01 06:07:38.050942 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:38.050901 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1f30d57-nz2b6"] Nov 01 06:07:38.101156 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:38.101121 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1f30d57-nz2b6"] Nov 01 06:07:39.515300 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:39.515267 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82f07425-72e6-43f1-9f54-fb18aef7a7a6" path="/var/lib/kubelet/pods/82f07425-72e6-43f1-9f54-fb18aef7a7a6/volumes" Nov 01 06:07:39.975948 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:07:39.975854 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:07:43.049607 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:43.049573 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfcfb1b2-qxgrb"] Nov 01 06:07:43.099638 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:43.099607 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfcfb1b2-qxgrb"] Nov 01 06:07:43.515839 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:43.515807 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="311f01ab-e4e0-4661-bd54-256ff29f3d39" path="/var/lib/kubelet/pods/311f01ab-e4e0-4661-bd54-256ff29f3d39/volumes" Nov 01 06:07:48.048942 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:48.048909 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bddf4e87-zcpb9"] Nov 01 06:07:48.099171 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:48.099137 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bddf4e87-zcpb9"] Nov 01 06:07:49.514375 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:49.514338 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5" path="/var/lib/kubelet/pods/0b9e2bbb-1d47-4cc4-bb0c-f50e6795e9e5/volumes" Nov 01 06:07:49.545950 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:49.545924 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e15fe852-pcqk4"] Nov 01 06:07:49.546420 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:49.546408 5040 cpu_manager.go:414] "containerMap: removing container" podUID="64837f7d-6b63-40de-a034-35701ceda44f" containerName="revoke-keys" Nov 01 06:07:49.546467 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:49.546421 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64837f7d-6b63-40de-a034-35701ceda44f" containerName="revoke-keys" Nov 01 06:07:49.546571 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:49.546560 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="64837f7d-6b63-40de-a034-35701ceda44f" containerName="revoke-keys" Nov 01 06:07:49.551216 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:49.551198 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15fe852-pcqk4" Nov 01 06:07:49.608777 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:49.608731 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e15fe852-pcqk4"] Nov 01 06:07:49.652936 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:49.652898 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:07:49.653128 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:49.652998 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:07:49.665458 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:49.665421 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/200b968b-65f6-4f89-9132-9caf0ff4e291-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e15fe852-pcqk4\" (UID: \"200b968b-65f6-4f89-9132-9caf0ff4e291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15fe852-pcqk4" Nov 01 06:07:49.665609 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:49.665466 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/200b968b-65f6-4f89-9132-9caf0ff4e291-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e15fe852-pcqk4\" (UID: \"200b968b-65f6-4f89-9132-9caf0ff4e291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15fe852-pcqk4" Nov 01 06:07:49.665698 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:49.665683 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdws4\" (UniqueName: \"kubernetes.io/projected/200b968b-65f6-4f89-9132-9caf0ff4e291-kube-api-access-cdws4\") pod \"maas-api-revoke-sub-e15fe852-pcqk4\" (UID: \"200b968b-65f6-4f89-9132-9caf0ff4e291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15fe852-pcqk4" Nov 01 06:07:49.692209 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:49.692181 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:07:49.703331 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:49.703308 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:07:49.767194 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:49.767115 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/200b968b-65f6-4f89-9132-9caf0ff4e291-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e15fe852-pcqk4\" (UID: \"200b968b-65f6-4f89-9132-9caf0ff4e291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15fe852-pcqk4" Nov 01 06:07:49.767194 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:49.767150 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/200b968b-65f6-4f89-9132-9caf0ff4e291-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e15fe852-pcqk4\" (UID: \"200b968b-65f6-4f89-9132-9caf0ff4e291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15fe852-pcqk4" Nov 01 06:07:49.767417 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:49.767194 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdws4\" (UniqueName: \"kubernetes.io/projected/200b968b-65f6-4f89-9132-9caf0ff4e291-kube-api-access-cdws4\") pod \"maas-api-revoke-sub-e15fe852-pcqk4\" (UID: \"200b968b-65f6-4f89-9132-9caf0ff4e291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15fe852-pcqk4" Nov 01 06:07:49.767929 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:49.767900 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/200b968b-65f6-4f89-9132-9caf0ff4e291-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e15fe852-pcqk4\" (UID: \"200b968b-65f6-4f89-9132-9caf0ff4e291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15fe852-pcqk4" Nov 01 06:07:49.769457 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:49.769441 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/200b968b-65f6-4f89-9132-9caf0ff4e291-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e15fe852-pcqk4\" (UID: \"200b968b-65f6-4f89-9132-9caf0ff4e291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15fe852-pcqk4" Nov 01 06:07:49.828990 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:49.828954 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdws4\" (UniqueName: \"kubernetes.io/projected/200b968b-65f6-4f89-9132-9caf0ff4e291-kube-api-access-cdws4\") pod \"maas-api-revoke-sub-e15fe852-pcqk4\" (UID: \"200b968b-65f6-4f89-9132-9caf0ff4e291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15fe852-pcqk4" Nov 01 06:07:49.864968 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:49.864929 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15fe852-pcqk4" Nov 01 06:07:50.253673 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:07:50.253639 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod200b968b_65f6_4f89_9132_9caf0ff4e291.slice/crio-d8c71cb2b96bcf37369fb62466cd9455b23262254035582c6fad63513da936ad WatchSource:0}: Error finding container d8c71cb2b96bcf37369fb62466cd9455b23262254035582c6fad63513da936ad: Status 404 returned error can't find the container with id d8c71cb2b96bcf37369fb62466cd9455b23262254035582c6fad63513da936ad Nov 01 06:07:50.253797 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:50.253780 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e15fe852-pcqk4"] Nov 01 06:07:50.454592 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:50.454474 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15fe852-pcqk4" event={"ID":"200b968b-65f6-4f89-9132-9caf0ff4e291","Type":"ContainerStarted","Data":"d8c71cb2b96bcf37369fb62466cd9455b23262254035582c6fad63513da936ad"} Nov 01 06:07:51.463708 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:51.463674 5040 generic.go:356] "Generic (PLEG): container finished" podID="200b968b-65f6-4f89-9132-9caf0ff4e291" containerID="04b4d41d2bd0187f7b8127e6904f2dad50dd7c281c0d5ae35e7d00f6b77dd70e" exitCode=0 Nov 01 06:07:51.464116 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:51.463758 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15fe852-pcqk4" event={"ID":"200b968b-65f6-4f89-9132-9caf0ff4e291","Type":"ContainerDied","Data":"04b4d41d2bd0187f7b8127e6904f2dad50dd7c281c0d5ae35e7d00f6b77dd70e"} Nov 01 06:07:52.677617 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:52.677590 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15fe852-pcqk4" Nov 01 06:07:52.805421 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:52.805381 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/200b968b-65f6-4f89-9132-9caf0ff4e291-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/200b968b-65f6-4f89-9132-9caf0ff4e291-maas-api-xks-ca\") pod \"200b968b-65f6-4f89-9132-9caf0ff4e291\" (UID: \"200b968b-65f6-4f89-9132-9caf0ff4e291\") " Nov 01 06:07:52.805623 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:52.805552 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/200b968b-65f6-4f89-9132-9caf0ff4e291-kube-api-access-cdws4\" (UniqueName: \"kubernetes.io/projected/200b968b-65f6-4f89-9132-9caf0ff4e291-kube-api-access-cdws4\") pod \"200b968b-65f6-4f89-9132-9caf0ff4e291\" (UID: \"200b968b-65f6-4f89-9132-9caf0ff4e291\") " Nov 01 06:07:52.805623 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:52.805616 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/200b968b-65f6-4f89-9132-9caf0ff4e291-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/200b968b-65f6-4f89-9132-9caf0ff4e291-maas-api-openshift-ca\") pod \"200b968b-65f6-4f89-9132-9caf0ff4e291\" (UID: \"200b968b-65f6-4f89-9132-9caf0ff4e291\") " Nov 01 06:07:52.806055 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:52.806023 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/200b968b-65f6-4f89-9132-9caf0ff4e291-maas-api-openshift-ca" pod "200b968b-65f6-4f89-9132-9caf0ff4e291" (UID: "200b968b-65f6-4f89-9132-9caf0ff4e291"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:07:52.808106 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:52.808081 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/200b968b-65f6-4f89-9132-9caf0ff4e291-maas-api-xks-ca" pod "200b968b-65f6-4f89-9132-9caf0ff4e291" (UID: "200b968b-65f6-4f89-9132-9caf0ff4e291"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:07:52.808314 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:52.808296 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/200b968b-65f6-4f89-9132-9caf0ff4e291-kube-api-access-cdws4" pod "200b968b-65f6-4f89-9132-9caf0ff4e291" (UID: "200b968b-65f6-4f89-9132-9caf0ff4e291"). InnerVolumeSpecName "kube-api-access-cdws4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:07:52.907152 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:52.907118 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdws4\" (UniqueName: \"kubernetes.io/projected/200b968b-65f6-4f89-9132-9caf0ff4e291-kube-api-access-cdws4\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:07:52.907152 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:52.907147 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/200b968b-65f6-4f89-9132-9caf0ff4e291-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:07:52.907152 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:52.907157 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/200b968b-65f6-4f89-9132-9caf0ff4e291-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:07:53.484967 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:53.484938 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15fe852-pcqk4" Nov 01 06:07:53.485141 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:53.484979 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15fe852-pcqk4" event={"ID":"200b968b-65f6-4f89-9132-9caf0ff4e291","Type":"ContainerDied","Data":"d8c71cb2b96bcf37369fb62466cd9455b23262254035582c6fad63513da936ad"} Nov 01 06:07:53.485141 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:53.485025 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8c71cb2b96bcf37369fb62466cd9455b23262254035582c6fad63513da936ad" Nov 01 06:07:57.059764 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:57.059724 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df8051bb-2cnn2"] Nov 01 06:07:57.060291 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:57.060276 5040 cpu_manager.go:414] "containerMap: removing container" podUID="200b968b-65f6-4f89-9132-9caf0ff4e291" containerName="revoke-keys" Nov 01 06:07:57.060332 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:57.060292 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="200b968b-65f6-4f89-9132-9caf0ff4e291" containerName="revoke-keys" Nov 01 06:07:57.060397 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:57.060388 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="200b968b-65f6-4f89-9132-9caf0ff4e291" containerName="revoke-keys" Nov 01 06:07:57.065189 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:57.065170 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8051bb-2cnn2" Nov 01 06:07:57.121958 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:57.121926 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df8051bb-2cnn2"] Nov 01 06:07:57.156578 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:57.156546 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl6jx\" (UniqueName: \"kubernetes.io/projected/ae460a49-f48c-48a9-b0be-a4d5d096411f-kube-api-access-wl6jx\") pod \"maas-api-revoke-sub-df8051bb-2cnn2\" (UID: \"ae460a49-f48c-48a9-b0be-a4d5d096411f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8051bb-2cnn2" Nov 01 06:07:57.156719 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:57.156621 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae460a49-f48c-48a9-b0be-a4d5d096411f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df8051bb-2cnn2\" (UID: \"ae460a49-f48c-48a9-b0be-a4d5d096411f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8051bb-2cnn2" Nov 01 06:07:57.156779 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:57.156729 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae460a49-f48c-48a9-b0be-a4d5d096411f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df8051bb-2cnn2\" (UID: \"ae460a49-f48c-48a9-b0be-a4d5d096411f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8051bb-2cnn2" Nov 01 06:07:57.167035 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:57.167014 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:07:57.168226 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:57.168205 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:07:57.168226 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:57.168206 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:07:57.168383 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:57.168211 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:07:57.257579 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:57.257546 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl6jx\" (UniqueName: \"kubernetes.io/projected/ae460a49-f48c-48a9-b0be-a4d5d096411f-kube-api-access-wl6jx\") pod \"maas-api-revoke-sub-df8051bb-2cnn2\" (UID: \"ae460a49-f48c-48a9-b0be-a4d5d096411f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8051bb-2cnn2" Nov 01 06:07:57.257780 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:57.257608 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae460a49-f48c-48a9-b0be-a4d5d096411f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df8051bb-2cnn2\" (UID: \"ae460a49-f48c-48a9-b0be-a4d5d096411f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8051bb-2cnn2" Nov 01 06:07:57.257780 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:57.257642 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae460a49-f48c-48a9-b0be-a4d5d096411f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df8051bb-2cnn2\" (UID: \"ae460a49-f48c-48a9-b0be-a4d5d096411f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8051bb-2cnn2" Nov 01 06:07:57.258315 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:57.258288 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae460a49-f48c-48a9-b0be-a4d5d096411f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df8051bb-2cnn2\" (UID: \"ae460a49-f48c-48a9-b0be-a4d5d096411f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8051bb-2cnn2" Nov 01 06:07:57.260028 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:57.260005 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae460a49-f48c-48a9-b0be-a4d5d096411f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df8051bb-2cnn2\" (UID: \"ae460a49-f48c-48a9-b0be-a4d5d096411f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8051bb-2cnn2" Nov 01 06:07:57.319440 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:57.319356 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl6jx\" (UniqueName: \"kubernetes.io/projected/ae460a49-f48c-48a9-b0be-a4d5d096411f-kube-api-access-wl6jx\") pod \"maas-api-revoke-sub-df8051bb-2cnn2\" (UID: \"ae460a49-f48c-48a9-b0be-a4d5d096411f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8051bb-2cnn2" Nov 01 06:07:57.379530 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:57.379495 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8051bb-2cnn2" Nov 01 06:07:57.774116 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:07:57.774084 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae460a49_f48c_48a9_b0be_a4d5d096411f.slice/crio-ad4ea8739309a7938a3c460dff775ab085ca5fca9d04d482d036da7f6b688abd WatchSource:0}: Error finding container ad4ea8739309a7938a3c460dff775ab085ca5fca9d04d482d036da7f6b688abd: Status 404 returned error can't find the container with id ad4ea8739309a7938a3c460dff775ab085ca5fca9d04d482d036da7f6b688abd Nov 01 06:07:57.775081 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:57.775057 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df8051bb-2cnn2"] Nov 01 06:07:58.523730 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:58.523692 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8051bb-2cnn2" event={"ID":"ae460a49-f48c-48a9-b0be-a4d5d096411f","Type":"ContainerStarted","Data":"4374b0a4a8bb149f84a283cc8a5fdd7601a58d3e2fe39d7a81ab444f30bf918c"} Nov 01 06:07:58.524148 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:58.523756 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8051bb-2cnn2" event={"ID":"ae460a49-f48c-48a9-b0be-a4d5d096411f","Type":"ContainerStarted","Data":"ad4ea8739309a7938a3c460dff775ab085ca5fca9d04d482d036da7f6b688abd"} Nov 01 06:07:58.644188 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:58.644120 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8051bb-2cnn2" podStartSLOduration=1.644106318 podStartE2EDuration="1.644106318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:07:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:07:58.639938475 +0000 UTC m=+1357.295055644" watchObservedRunningTime="2026-11-01 06:07:58.644106318 +0000 UTC m=+1357.299223484" Nov 01 06:07:59.533169 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:59.533128 5040 generic.go:356] "Generic (PLEG): container finished" podID="ae460a49-f48c-48a9-b0be-a4d5d096411f" containerID="4374b0a4a8bb149f84a283cc8a5fdd7601a58d3e2fe39d7a81ab444f30bf918c" exitCode=0 Nov 01 06:07:59.533609 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:07:59.533216 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8051bb-2cnn2" event={"ID":"ae460a49-f48c-48a9-b0be-a4d5d096411f","Type":"ContainerDied","Data":"4374b0a4a8bb149f84a283cc8a5fdd7601a58d3e2fe39d7a81ab444f30bf918c"} Nov 01 06:08:00.048958 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:00.048923 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2e981c2-68tqn"] Nov 01 06:08:00.052890 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:00.052863 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2e981c2-68tqn"] Nov 01 06:08:00.371055 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:00.370976 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d981b767-hx96n"] Nov 01 06:08:00.375556 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:00.375532 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d981b767-hx96n" Nov 01 06:08:00.469953 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:00.469914 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d981b767-hx96n"] Nov 01 06:08:00.492172 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:00.492137 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2ee8964-c194-4d28-94d1-7518f2e69e15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d981b767-hx96n\" (UID: \"a2ee8964-c194-4d28-94d1-7518f2e69e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d981b767-hx96n" Nov 01 06:08:00.492360 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:00.492226 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2ee8964-c194-4d28-94d1-7518f2e69e15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d981b767-hx96n\" (UID: \"a2ee8964-c194-4d28-94d1-7518f2e69e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d981b767-hx96n" Nov 01 06:08:00.492360 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:00.492327 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc9xx\" (UniqueName: \"kubernetes.io/projected/a2ee8964-c194-4d28-94d1-7518f2e69e15-kube-api-access-pc9xx\") pod \"maas-api-revoke-sub-d981b767-hx96n\" (UID: \"a2ee8964-c194-4d28-94d1-7518f2e69e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d981b767-hx96n" Nov 01 06:08:00.593957 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:00.593926 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2ee8964-c194-4d28-94d1-7518f2e69e15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d981b767-hx96n\" (UID: \"a2ee8964-c194-4d28-94d1-7518f2e69e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d981b767-hx96n" Nov 01 06:08:00.594441 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:00.593986 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2ee8964-c194-4d28-94d1-7518f2e69e15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d981b767-hx96n\" (UID: \"a2ee8964-c194-4d28-94d1-7518f2e69e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d981b767-hx96n" Nov 01 06:08:00.594441 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:00.594029 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc9xx\" (UniqueName: \"kubernetes.io/projected/a2ee8964-c194-4d28-94d1-7518f2e69e15-kube-api-access-pc9xx\") pod \"maas-api-revoke-sub-d981b767-hx96n\" (UID: \"a2ee8964-c194-4d28-94d1-7518f2e69e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d981b767-hx96n" Nov 01 06:08:00.594759 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:00.594737 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2ee8964-c194-4d28-94d1-7518f2e69e15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d981b767-hx96n\" (UID: \"a2ee8964-c194-4d28-94d1-7518f2e69e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d981b767-hx96n" Nov 01 06:08:00.596589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:00.596565 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2ee8964-c194-4d28-94d1-7518f2e69e15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d981b767-hx96n\" (UID: \"a2ee8964-c194-4d28-94d1-7518f2e69e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d981b767-hx96n" Nov 01 06:08:00.675309 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:00.675229 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc9xx\" (UniqueName: \"kubernetes.io/projected/a2ee8964-c194-4d28-94d1-7518f2e69e15-kube-api-access-pc9xx\") pod \"maas-api-revoke-sub-d981b767-hx96n\" (UID: \"a2ee8964-c194-4d28-94d1-7518f2e69e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d981b767-hx96n" Nov 01 06:08:00.691885 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:00.691852 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d981b767-hx96n" Nov 01 06:08:00.726218 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:00.726189 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8051bb-2cnn2" Nov 01 06:08:00.796303 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:00.796270 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae460a49-f48c-48a9-b0be-a4d5d096411f-kube-api-access-wl6jx\" (UniqueName: \"kubernetes.io/projected/ae460a49-f48c-48a9-b0be-a4d5d096411f-kube-api-access-wl6jx\") pod \"ae460a49-f48c-48a9-b0be-a4d5d096411f\" (UID: \"ae460a49-f48c-48a9-b0be-a4d5d096411f\") " Nov 01 06:08:00.796510 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:00.796381 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae460a49-f48c-48a9-b0be-a4d5d096411f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae460a49-f48c-48a9-b0be-a4d5d096411f-maas-api-xks-ca\") pod \"ae460a49-f48c-48a9-b0be-a4d5d096411f\" (UID: \"ae460a49-f48c-48a9-b0be-a4d5d096411f\") " Nov 01 06:08:00.796510 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:00.796443 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae460a49-f48c-48a9-b0be-a4d5d096411f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae460a49-f48c-48a9-b0be-a4d5d096411f-maas-api-openshift-ca\") pod \"ae460a49-f48c-48a9-b0be-a4d5d096411f\" (UID: \"ae460a49-f48c-48a9-b0be-a4d5d096411f\") " Nov 01 06:08:00.796983 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:00.796934 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae460a49-f48c-48a9-b0be-a4d5d096411f-maas-api-openshift-ca" pod "ae460a49-f48c-48a9-b0be-a4d5d096411f" (UID: "ae460a49-f48c-48a9-b0be-a4d5d096411f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:08:00.799571 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:00.799543 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae460a49-f48c-48a9-b0be-a4d5d096411f-maas-api-xks-ca" pod "ae460a49-f48c-48a9-b0be-a4d5d096411f" (UID: "ae460a49-f48c-48a9-b0be-a4d5d096411f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:08:00.800362 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:00.800334 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae460a49-f48c-48a9-b0be-a4d5d096411f-kube-api-access-wl6jx" pod "ae460a49-f48c-48a9-b0be-a4d5d096411f" (UID: "ae460a49-f48c-48a9-b0be-a4d5d096411f"). InnerVolumeSpecName "kube-api-access-wl6jx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:08:00.897985 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:00.897948 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl6jx\" (UniqueName: \"kubernetes.io/projected/ae460a49-f48c-48a9-b0be-a4d5d096411f-kube-api-access-wl6jx\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:08:00.897985 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:00.897981 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae460a49-f48c-48a9-b0be-a4d5d096411f-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:08:00.897985 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:00.897991 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae460a49-f48c-48a9-b0be-a4d5d096411f-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:08:01.082232 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:08:01.082199 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ee8964_c194_4d28_94d1_7518f2e69e15.slice/crio-af95ce402135828ddaa0b4f8c0ffb85d383e7cab9f87276d2f683d2c13431353 WatchSource:0}: Error finding container af95ce402135828ddaa0b4f8c0ffb85d383e7cab9f87276d2f683d2c13431353: Status 404 returned error can't find the container with id af95ce402135828ddaa0b4f8c0ffb85d383e7cab9f87276d2f683d2c13431353 Nov 01 06:08:01.084800 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:01.084772 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d981b767-hx96n"] Nov 01 06:08:01.522505 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:01.521640 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71284f4a-dfa0-410c-9cf2-f5753d6f21e8" path="/var/lib/kubelet/pods/71284f4a-dfa0-410c-9cf2-f5753d6f21e8/volumes" Nov 01 06:08:01.556465 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:01.556432 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8051bb-2cnn2" Nov 01 06:08:01.556651 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:01.556429 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8051bb-2cnn2" event={"ID":"ae460a49-f48c-48a9-b0be-a4d5d096411f","Type":"ContainerDied","Data":"ad4ea8739309a7938a3c460dff775ab085ca5fca9d04d482d036da7f6b688abd"} Nov 01 06:08:01.556749 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:01.556653 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad4ea8739309a7938a3c460dff775ab085ca5fca9d04d482d036da7f6b688abd" Nov 01 06:08:01.558418 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:01.558385 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d981b767-hx96n" event={"ID":"a2ee8964-c194-4d28-94d1-7518f2e69e15","Type":"ContainerStarted","Data":"af95ce402135828ddaa0b4f8c0ffb85d383e7cab9f87276d2f683d2c13431353"} Nov 01 06:08:02.612441 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:02.597138 5040 generic.go:356] "Generic (PLEG): container finished" podID="a2ee8964-c194-4d28-94d1-7518f2e69e15" containerID="be8b20a6ec342f0c07a0b20bc82f256d83c3f641400c6fbcd6a10dc3ec29be53" exitCode=0 Nov 01 06:08:02.612441 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:02.597388 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d981b767-hx96n" event={"ID":"a2ee8964-c194-4d28-94d1-7518f2e69e15","Type":"ContainerDied","Data":"be8b20a6ec342f0c07a0b20bc82f256d83c3f641400c6fbcd6a10dc3ec29be53"} Nov 01 06:08:03.788660 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:03.788587 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d981b767-hx96n" Nov 01 06:08:03.898961 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:03.898881 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2ee8964-c194-4d28-94d1-7518f2e69e15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2ee8964-c194-4d28-94d1-7518f2e69e15-maas-api-openshift-ca\") pod \"a2ee8964-c194-4d28-94d1-7518f2e69e15\" (UID: \"a2ee8964-c194-4d28-94d1-7518f2e69e15\") " Nov 01 06:08:03.898961 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:03.898956 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2ee8964-c194-4d28-94d1-7518f2e69e15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2ee8964-c194-4d28-94d1-7518f2e69e15-maas-api-xks-ca\") pod \"a2ee8964-c194-4d28-94d1-7518f2e69e15\" (UID: \"a2ee8964-c194-4d28-94d1-7518f2e69e15\") " Nov 01 06:08:03.899256 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:03.899230 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2ee8964-c194-4d28-94d1-7518f2e69e15-kube-api-access-pc9xx\" (UniqueName: \"kubernetes.io/projected/a2ee8964-c194-4d28-94d1-7518f2e69e15-kube-api-access-pc9xx\") pod \"a2ee8964-c194-4d28-94d1-7518f2e69e15\" (UID: \"a2ee8964-c194-4d28-94d1-7518f2e69e15\") " Nov 01 06:08:03.899359 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:03.899250 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ee8964-c194-4d28-94d1-7518f2e69e15-maas-api-openshift-ca" pod "a2ee8964-c194-4d28-94d1-7518f2e69e15" (UID: "a2ee8964-c194-4d28-94d1-7518f2e69e15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:08:03.899654 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:03.899637 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2ee8964-c194-4d28-94d1-7518f2e69e15-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:08:03.901014 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:03.900993 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2ee8964-c194-4d28-94d1-7518f2e69e15-maas-api-xks-ca" pod "a2ee8964-c194-4d28-94d1-7518f2e69e15" (UID: "a2ee8964-c194-4d28-94d1-7518f2e69e15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:08:03.901327 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:03.901307 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ee8964-c194-4d28-94d1-7518f2e69e15-kube-api-access-pc9xx" pod "a2ee8964-c194-4d28-94d1-7518f2e69e15" (UID: "a2ee8964-c194-4d28-94d1-7518f2e69e15"). InnerVolumeSpecName "kube-api-access-pc9xx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:08:04.001230 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:04.001192 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc9xx\" (UniqueName: \"kubernetes.io/projected/a2ee8964-c194-4d28-94d1-7518f2e69e15-kube-api-access-pc9xx\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:08:04.001230 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:04.001230 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2ee8964-c194-4d28-94d1-7518f2e69e15-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:08:04.616527 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:04.616501 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d981b767-hx96n" Nov 01 06:08:04.616527 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:04.616510 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d981b767-hx96n" event={"ID":"a2ee8964-c194-4d28-94d1-7518f2e69e15","Type":"ContainerDied","Data":"af95ce402135828ddaa0b4f8c0ffb85d383e7cab9f87276d2f683d2c13431353"} Nov 01 06:08:04.616755 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:04.616565 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af95ce402135828ddaa0b4f8c0ffb85d383e7cab9f87276d2f683d2c13431353" Nov 01 06:08:05.048508 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:05.048446 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daeedd40-96gjv"] Nov 01 06:08:05.052679 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:05.052650 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daeedd40-96gjv"] Nov 01 06:08:05.516757 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:05.516725 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ecf787f-f866-4daa-9354-db17829557e1" path="/var/lib/kubelet/pods/5ecf787f-f866-4daa-9354-db17829557e1/volumes" Nov 01 06:08:09.973374 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:08:09.973337 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:08:12.480414 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:12.480381 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3721c766-plfmj"] Nov 01 06:08:12.480900 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:12.480886 5040 cpu_manager.go:414] "containerMap: removing container" podUID="ae460a49-f48c-48a9-b0be-a4d5d096411f" containerName="revoke-keys" Nov 01 06:08:12.480943 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:12.480900 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae460a49-f48c-48a9-b0be-a4d5d096411f" containerName="revoke-keys" Nov 01 06:08:12.480943 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:12.480941 5040 cpu_manager.go:414] "containerMap: removing container" podUID="a2ee8964-c194-4d28-94d1-7518f2e69e15" containerName="revoke-keys" Nov 01 06:08:12.481019 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:12.480947 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ee8964-c194-4d28-94d1-7518f2e69e15" containerName="revoke-keys" Nov 01 06:08:12.481050 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:12.481041 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2ee8964-c194-4d28-94d1-7518f2e69e15" containerName="revoke-keys" Nov 01 06:08:12.481083 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:12.481051 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae460a49-f48c-48a9-b0be-a4d5d096411f" containerName="revoke-keys" Nov 01 06:08:12.484566 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:12.484551 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3721c766-plfmj" Nov 01 06:08:12.542329 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:12.542296 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3721c766-plfmj"] Nov 01 06:08:12.586375 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:12.586340 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:08:12.586574 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:12.586386 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:08:12.587890 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:12.587875 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:08:12.587952 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:12.587898 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:08:12.609944 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:12.609914 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mpvp\" (UniqueName: \"kubernetes.io/projected/69e53ef2-a284-4996-a030-195fc5d509fa-kube-api-access-4mpvp\") pod \"maas-api-revoke-sub-3721c766-plfmj\" (UID: \"69e53ef2-a284-4996-a030-195fc5d509fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3721c766-plfmj" Nov 01 06:08:12.610159 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:12.610141 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69e53ef2-a284-4996-a030-195fc5d509fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3721c766-plfmj\" (UID: \"69e53ef2-a284-4996-a030-195fc5d509fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3721c766-plfmj" Nov 01 06:08:12.610217 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:12.610187 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69e53ef2-a284-4996-a030-195fc5d509fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3721c766-plfmj\" (UID: \"69e53ef2-a284-4996-a030-195fc5d509fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3721c766-plfmj" Nov 01 06:08:12.711595 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:12.711564 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mpvp\" (UniqueName: \"kubernetes.io/projected/69e53ef2-a284-4996-a030-195fc5d509fa-kube-api-access-4mpvp\") pod \"maas-api-revoke-sub-3721c766-plfmj\" (UID: \"69e53ef2-a284-4996-a030-195fc5d509fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3721c766-plfmj" Nov 01 06:08:12.711767 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:12.711672 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69e53ef2-a284-4996-a030-195fc5d509fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3721c766-plfmj\" (UID: \"69e53ef2-a284-4996-a030-195fc5d509fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3721c766-plfmj" Nov 01 06:08:12.711767 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:12.711709 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69e53ef2-a284-4996-a030-195fc5d509fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3721c766-plfmj\" (UID: \"69e53ef2-a284-4996-a030-195fc5d509fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3721c766-plfmj" Nov 01 06:08:12.712674 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:12.712634 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69e53ef2-a284-4996-a030-195fc5d509fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3721c766-plfmj\" (UID: \"69e53ef2-a284-4996-a030-195fc5d509fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3721c766-plfmj" Nov 01 06:08:12.714462 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:12.714436 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69e53ef2-a284-4996-a030-195fc5d509fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3721c766-plfmj\" (UID: \"69e53ef2-a284-4996-a030-195fc5d509fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3721c766-plfmj" Nov 01 06:08:12.773468 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:12.773443 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mpvp\" (UniqueName: \"kubernetes.io/projected/69e53ef2-a284-4996-a030-195fc5d509fa-kube-api-access-4mpvp\") pod \"maas-api-revoke-sub-3721c766-plfmj\" (UID: \"69e53ef2-a284-4996-a030-195fc5d509fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3721c766-plfmj" Nov 01 06:08:12.797975 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:12.797951 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3721c766-plfmj" Nov 01 06:08:13.181570 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:08:13.181533 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69e53ef2_a284_4996_a030_195fc5d509fa.slice/crio-b0a76df1c71bb38fa96d4f2fca7b9b26fb878c01b25d033851d9a78d479746ad WatchSource:0}: Error finding container b0a76df1c71bb38fa96d4f2fca7b9b26fb878c01b25d033851d9a78d479746ad: Status 404 returned error can't find the container with id b0a76df1c71bb38fa96d4f2fca7b9b26fb878c01b25d033851d9a78d479746ad Nov 01 06:08:13.181703 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:13.181611 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3721c766-plfmj"] Nov 01 06:08:13.687378 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:13.687339 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3721c766-plfmj" event={"ID":"69e53ef2-a284-4996-a030-195fc5d509fa","Type":"ContainerStarted","Data":"18d971d620c08f44fba39210a7d87318730e80c4066e3c7ebcf1d28367c964c0"} Nov 01 06:08:13.687789 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:13.687395 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3721c766-plfmj" event={"ID":"69e53ef2-a284-4996-a030-195fc5d509fa","Type":"ContainerStarted","Data":"b0a76df1c71bb38fa96d4f2fca7b9b26fb878c01b25d033851d9a78d479746ad"} Nov 01 06:08:13.814071 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:13.813937 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3721c766-plfmj" podStartSLOduration=1.813917083 podStartE2EDuration="1.813917083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:08:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:08:13.809947804 +0000 UTC m=+1372.465064971" watchObservedRunningTime="2026-11-01 06:08:13.813917083 +0000 UTC m=+1372.469034254" Nov 01 06:08:14.697568 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:14.697529 5040 generic.go:356] "Generic (PLEG): container finished" podID="69e53ef2-a284-4996-a030-195fc5d509fa" containerID="18d971d620c08f44fba39210a7d87318730e80c4066e3c7ebcf1d28367c964c0" exitCode=0 Nov 01 06:08:14.697954 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:14.697612 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3721c766-plfmj" event={"ID":"69e53ef2-a284-4996-a030-195fc5d509fa","Type":"ContainerDied","Data":"18d971d620c08f44fba39210a7d87318730e80c4066e3c7ebcf1d28367c964c0"} Nov 01 06:08:15.049410 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:15.049370 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eabef2e-w47sw"] Nov 01 06:08:15.053386 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:15.053360 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eabef2e-w47sw"] Nov 01 06:08:15.515335 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:15.515304 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b433b2b9-e929-436d-ae27-9d42a76cbb40" path="/var/lib/kubelet/pods/b433b2b9-e929-436d-ae27-9d42a76cbb40/volumes" Nov 01 06:08:15.861681 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:15.861658 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3721c766-plfmj" Nov 01 06:08:15.949044 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:15.949008 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69e53ef2-a284-4996-a030-195fc5d509fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69e53ef2-a284-4996-a030-195fc5d509fa-maas-api-openshift-ca\") pod \"69e53ef2-a284-4996-a030-195fc5d509fa\" (UID: \"69e53ef2-a284-4996-a030-195fc5d509fa\") " Nov 01 06:08:15.949237 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:15.949061 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69e53ef2-a284-4996-a030-195fc5d509fa-kube-api-access-4mpvp\" (UniqueName: \"kubernetes.io/projected/69e53ef2-a284-4996-a030-195fc5d509fa-kube-api-access-4mpvp\") pod \"69e53ef2-a284-4996-a030-195fc5d509fa\" (UID: \"69e53ef2-a284-4996-a030-195fc5d509fa\") " Nov 01 06:08:15.949237 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:15.949152 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69e53ef2-a284-4996-a030-195fc5d509fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69e53ef2-a284-4996-a030-195fc5d509fa-maas-api-xks-ca\") pod \"69e53ef2-a284-4996-a030-195fc5d509fa\" (UID: \"69e53ef2-a284-4996-a030-195fc5d509fa\") " Nov 01 06:08:15.949427 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:15.949401 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69e53ef2-a284-4996-a030-195fc5d509fa-maas-api-openshift-ca" pod "69e53ef2-a284-4996-a030-195fc5d509fa" (UID: "69e53ef2-a284-4996-a030-195fc5d509fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:08:15.949632 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:15.949615 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69e53ef2-a284-4996-a030-195fc5d509fa-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:08:15.951161 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:15.951136 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69e53ef2-a284-4996-a030-195fc5d509fa-maas-api-xks-ca" pod "69e53ef2-a284-4996-a030-195fc5d509fa" (UID: "69e53ef2-a284-4996-a030-195fc5d509fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:08:15.951244 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:15.951222 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69e53ef2-a284-4996-a030-195fc5d509fa-kube-api-access-4mpvp" pod "69e53ef2-a284-4996-a030-195fc5d509fa" (UID: "69e53ef2-a284-4996-a030-195fc5d509fa"). InnerVolumeSpecName "kube-api-access-4mpvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:08:16.051194 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:16.051139 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mpvp\" (UniqueName: \"kubernetes.io/projected/69e53ef2-a284-4996-a030-195fc5d509fa-kube-api-access-4mpvp\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:08:16.051194 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:16.051188 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69e53ef2-a284-4996-a030-195fc5d509fa-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:08:16.715646 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:16.715617 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3721c766-plfmj" Nov 01 06:08:16.715836 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:16.715651 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3721c766-plfmj" event={"ID":"69e53ef2-a284-4996-a030-195fc5d509fa","Type":"ContainerDied","Data":"b0a76df1c71bb38fa96d4f2fca7b9b26fb878c01b25d033851d9a78d479746ad"} Nov 01 06:08:16.715836 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:16.715700 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0a76df1c71bb38fa96d4f2fca7b9b26fb878c01b25d033851d9a78d479746ad" Nov 01 06:08:21.762785 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:21.762748 5040 scope.go:122] "RemoveContainer" containerID="c8314eb0927c9fb4effcc5ef444f95fbf819962f5615e060c17cf9b86acfb286" Nov 01 06:08:21.773194 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:21.773170 5040 scope.go:122] "RemoveContainer" containerID="f378d9633cc8088e84c2bf64e9b9841433a504ebbe5a4481f74d5fb440d2e121" Nov 01 06:08:27.728761 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:27.728723 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6608305-fjc8z"] Nov 01 06:08:27.729274 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:27.729259 5040 cpu_manager.go:414] "containerMap: removing container" podUID="69e53ef2-a284-4996-a030-195fc5d509fa" containerName="revoke-keys" Nov 01 06:08:27.729314 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:27.729276 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69e53ef2-a284-4996-a030-195fc5d509fa" containerName="revoke-keys" Nov 01 06:08:27.729391 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:27.729382 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="69e53ef2-a284-4996-a030-195fc5d509fa" containerName="revoke-keys" Nov 01 06:08:27.734530 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:27.734509 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6608305-fjc8z" Nov 01 06:08:27.778026 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:27.777986 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e58e9d0-1122-480f-b5ed-60cadb389f23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6608305-fjc8z\" (UID: \"3e58e9d0-1122-480f-b5ed-60cadb389f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6608305-fjc8z" Nov 01 06:08:27.778196 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:27.778053 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e58e9d0-1122-480f-b5ed-60cadb389f23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6608305-fjc8z\" (UID: \"3e58e9d0-1122-480f-b5ed-60cadb389f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6608305-fjc8z" Nov 01 06:08:27.778196 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:27.778138 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvhwp\" (UniqueName: \"kubernetes.io/projected/3e58e9d0-1122-480f-b5ed-60cadb389f23-kube-api-access-lvhwp\") pod \"maas-api-revoke-sub-b6608305-fjc8z\" (UID: \"3e58e9d0-1122-480f-b5ed-60cadb389f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6608305-fjc8z" Nov 01 06:08:27.793595 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:27.793557 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6608305-fjc8z"] Nov 01 06:08:27.834979 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:27.834945 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:08:27.835142 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:27.835027 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:08:27.835142 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:27.835088 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:08:27.835142 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:27.835110 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:08:27.879408 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:27.879376 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvhwp\" (UniqueName: \"kubernetes.io/projected/3e58e9d0-1122-480f-b5ed-60cadb389f23-kube-api-access-lvhwp\") pod \"maas-api-revoke-sub-b6608305-fjc8z\" (UID: \"3e58e9d0-1122-480f-b5ed-60cadb389f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6608305-fjc8z" Nov 01 06:08:27.879624 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:27.879584 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e58e9d0-1122-480f-b5ed-60cadb389f23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6608305-fjc8z\" (UID: \"3e58e9d0-1122-480f-b5ed-60cadb389f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6608305-fjc8z" Nov 01 06:08:27.879668 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:27.879638 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e58e9d0-1122-480f-b5ed-60cadb389f23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6608305-fjc8z\" (UID: \"3e58e9d0-1122-480f-b5ed-60cadb389f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6608305-fjc8z" Nov 01 06:08:27.880407 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:27.880381 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e58e9d0-1122-480f-b5ed-60cadb389f23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6608305-fjc8z\" (UID: \"3e58e9d0-1122-480f-b5ed-60cadb389f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6608305-fjc8z" Nov 01 06:08:27.882034 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:27.882018 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e58e9d0-1122-480f-b5ed-60cadb389f23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6608305-fjc8z\" (UID: \"3e58e9d0-1122-480f-b5ed-60cadb389f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6608305-fjc8z" Nov 01 06:08:27.941500 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:27.941455 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvhwp\" (UniqueName: \"kubernetes.io/projected/3e58e9d0-1122-480f-b5ed-60cadb389f23-kube-api-access-lvhwp\") pod \"maas-api-revoke-sub-b6608305-fjc8z\" (UID: \"3e58e9d0-1122-480f-b5ed-60cadb389f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6608305-fjc8z" Nov 01 06:08:28.048128 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:28.048092 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6608305-fjc8z" Nov 01 06:08:28.049318 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:28.049294 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea44eee9-vcqz6"] Nov 01 06:08:28.057905 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:28.054176 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea44eee9-vcqz6"] Nov 01 06:08:28.428709 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:08:28.428667 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e58e9d0_1122_480f_b5ed_60cadb389f23.slice/crio-cb4f8f0b859bb28dfb880e856b7241682c471c058fa18331137770cf83b81e4a WatchSource:0}: Error finding container cb4f8f0b859bb28dfb880e856b7241682c471c058fa18331137770cf83b81e4a: Status 404 returned error can't find the container with id cb4f8f0b859bb28dfb880e856b7241682c471c058fa18331137770cf83b81e4a Nov 01 06:08:28.428847 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:28.428778 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6608305-fjc8z"] Nov 01 06:08:28.851918 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:28.851883 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6608305-fjc8z" event={"ID":"3e58e9d0-1122-480f-b5ed-60cadb389f23","Type":"ContainerStarted","Data":"7ef8bf1798e27eacf827ebe3651ea967260c4aae4b14bf2ed92d2fdd81f98158"} Nov 01 06:08:28.852318 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:28.851936 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6608305-fjc8z" event={"ID":"3e58e9d0-1122-480f-b5ed-60cadb389f23","Type":"ContainerStarted","Data":"cb4f8f0b859bb28dfb880e856b7241682c471c058fa18331137770cf83b81e4a"} Nov 01 06:08:28.970859 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:28.970750 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6608305-fjc8z" podStartSLOduration=1.9707355039999999 podStartE2EDuration="1.970735504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:08:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:08:28.969955899 +0000 UTC m=+1387.625073067" watchObservedRunningTime="2026-11-01 06:08:28.970735504 +0000 UTC m=+1387.625852671" Nov 01 06:08:29.515741 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:29.515660 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f434da95-fe85-4951-af26-2e9ef793a51d" path="/var/lib/kubelet/pods/f434da95-fe85-4951-af26-2e9ef793a51d/volumes" Nov 01 06:08:29.860801 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:29.860717 5040 generic.go:356] "Generic (PLEG): container finished" podID="f41d65c3-acc5-4de4-a46f-18787ecf7925" containerID="6cb45f503b034a20a145d84cf4ab8255248b65e1a53078fdeae876d3ddbf30ee" exitCode=0 Nov 01 06:08:29.860801 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:29.860789 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fhpc7-7qkbj" event={"ID":"f41d65c3-acc5-4de4-a46f-18787ecf7925","Type":"ContainerDied","Data":"6cb45f503b034a20a145d84cf4ab8255248b65e1a53078fdeae876d3ddbf30ee"} Nov 01 06:08:29.862740 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:29.862706 5040 generic.go:356] "Generic (PLEG): container finished" podID="3e58e9d0-1122-480f-b5ed-60cadb389f23" containerID="7ef8bf1798e27eacf827ebe3651ea967260c4aae4b14bf2ed92d2fdd81f98158" exitCode=0 Nov 01 06:08:29.862864 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:29.862759 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6608305-fjc8z" event={"ID":"3e58e9d0-1122-480f-b5ed-60cadb389f23","Type":"ContainerDied","Data":"7ef8bf1798e27eacf827ebe3651ea967260c4aae4b14bf2ed92d2fdd81f98158"} Nov 01 06:08:31.081332 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:31.081293 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fhpc7-7qkbj" Nov 01 06:08:31.084312 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:31.084289 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6608305-fjc8z" Nov 01 06:08:31.216439 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:31.216351 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f41d65c3-acc5-4de4-a46f-18787ecf7925-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f41d65c3-acc5-4de4-a46f-18787ecf7925-service-ca-bundle\") pod \"f41d65c3-acc5-4de4-a46f-18787ecf7925\" (UID: \"f41d65c3-acc5-4de4-a46f-18787ecf7925\") " Nov 01 06:08:31.216439 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:31.216397 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f41d65c3-acc5-4de4-a46f-18787ecf7925-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f41d65c3-acc5-4de4-a46f-18787ecf7925-archives-path\") pod \"f41d65c3-acc5-4de4-a46f-18787ecf7925\" (UID: \"f41d65c3-acc5-4de4-a46f-18787ecf7925\") " Nov 01 06:08:31.216439 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:31.216416 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f41d65c3-acc5-4de4-a46f-18787ecf7925-kube-api-access-5vt7f\" (UniqueName: \"kubernetes.io/projected/f41d65c3-acc5-4de4-a46f-18787ecf7925-kube-api-access-5vt7f\") pod \"f41d65c3-acc5-4de4-a46f-18787ecf7925\" (UID: \"f41d65c3-acc5-4de4-a46f-18787ecf7925\") " Nov 01 06:08:31.216734 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:31.216469 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e58e9d0-1122-480f-b5ed-60cadb389f23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e58e9d0-1122-480f-b5ed-60cadb389f23-maas-api-xks-ca\") pod \"3e58e9d0-1122-480f-b5ed-60cadb389f23\" (UID: \"3e58e9d0-1122-480f-b5ed-60cadb389f23\") " Nov 01 06:08:31.216734 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:31.216512 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f41d65c3-acc5-4de4-a46f-18787ecf7925-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f41d65c3-acc5-4de4-a46f-18787ecf7925-trusted-ca-bundle\") pod \"f41d65c3-acc5-4de4-a46f-18787ecf7925\" (UID: \"f41d65c3-acc5-4de4-a46f-18787ecf7925\") " Nov 01 06:08:31.216734 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:31.216550 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e58e9d0-1122-480f-b5ed-60cadb389f23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e58e9d0-1122-480f-b5ed-60cadb389f23-maas-api-openshift-ca\") pod \"3e58e9d0-1122-480f-b5ed-60cadb389f23\" (UID: \"3e58e9d0-1122-480f-b5ed-60cadb389f23\") " Nov 01 06:08:31.216900 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:31.216790 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f41d65c3-acc5-4de4-a46f-18787ecf7925-archives-path" pod "f41d65c3-acc5-4de4-a46f-18787ecf7925" (UID: "f41d65c3-acc5-4de4-a46f-18787ecf7925"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:08:31.216974 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:31.216907 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f41d65c3-acc5-4de4-a46f-18787ecf7925-service-ca-bundle" pod "f41d65c3-acc5-4de4-a46f-18787ecf7925" (UID: "f41d65c3-acc5-4de4-a46f-18787ecf7925"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:08:31.217029 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:31.216994 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e58e9d0-1122-480f-b5ed-60cadb389f23-maas-api-openshift-ca" pod "3e58e9d0-1122-480f-b5ed-60cadb389f23" (UID: "3e58e9d0-1122-480f-b5ed-60cadb389f23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:08:31.217157 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:31.217124 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e58e9d0-1122-480f-b5ed-60cadb389f23-kube-api-access-lvhwp\" (UniqueName: \"kubernetes.io/projected/3e58e9d0-1122-480f-b5ed-60cadb389f23-kube-api-access-lvhwp\") pod \"3e58e9d0-1122-480f-b5ed-60cadb389f23\" (UID: \"3e58e9d0-1122-480f-b5ed-60cadb389f23\") " Nov 01 06:08:31.217245 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:31.217202 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f41d65c3-acc5-4de4-a46f-18787ecf7925-trusted-ca-bundle" pod "f41d65c3-acc5-4de4-a46f-18787ecf7925" (UID: "f41d65c3-acc5-4de4-a46f-18787ecf7925"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:08:31.217667 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:31.217631 5040 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f41d65c3-acc5-4de4-a46f-18787ecf7925-service-ca-bundle\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:08:31.217667 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:31.217660 5040 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f41d65c3-acc5-4de4-a46f-18787ecf7925-archives-path\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:08:31.217829 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:31.217677 5040 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f41d65c3-acc5-4de4-a46f-18787ecf7925-trusted-ca-bundle\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:08:31.217829 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:31.217711 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e58e9d0-1122-480f-b5ed-60cadb389f23-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:08:31.218911 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:31.218876 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e58e9d0-1122-480f-b5ed-60cadb389f23-maas-api-xks-ca" pod "3e58e9d0-1122-480f-b5ed-60cadb389f23" (UID: "3e58e9d0-1122-480f-b5ed-60cadb389f23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:08:31.219060 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:31.219043 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f41d65c3-acc5-4de4-a46f-18787ecf7925-kube-api-access-5vt7f" pod "f41d65c3-acc5-4de4-a46f-18787ecf7925" (UID: "f41d65c3-acc5-4de4-a46f-18787ecf7925"). InnerVolumeSpecName "kube-api-access-5vt7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:08:31.219185 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:31.219166 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e58e9d0-1122-480f-b5ed-60cadb389f23-kube-api-access-lvhwp" pod "3e58e9d0-1122-480f-b5ed-60cadb389f23" (UID: "3e58e9d0-1122-480f-b5ed-60cadb389f23"). InnerVolumeSpecName "kube-api-access-lvhwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:08:31.318875 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:31.318843 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e58e9d0-1122-480f-b5ed-60cadb389f23-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:08:31.318875 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:31.318871 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvhwp\" (UniqueName: \"kubernetes.io/projected/3e58e9d0-1122-480f-b5ed-60cadb389f23-kube-api-access-lvhwp\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:08:31.318875 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:31.318882 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vt7f\" (UniqueName: \"kubernetes.io/projected/f41d65c3-acc5-4de4-a46f-18787ecf7925-kube-api-access-5vt7f\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:08:31.880096 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:31.880012 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6608305-fjc8z" Nov 01 06:08:31.880096 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:31.880038 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6608305-fjc8z" event={"ID":"3e58e9d0-1122-480f-b5ed-60cadb389f23","Type":"ContainerDied","Data":"cb4f8f0b859bb28dfb880e856b7241682c471c058fa18331137770cf83b81e4a"} Nov 01 06:08:31.880096 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:31.880093 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb4f8f0b859bb28dfb880e856b7241682c471c058fa18331137770cf83b81e4a" Nov 01 06:08:31.881636 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:31.881615 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fhpc7-7qkbj" Nov 01 06:08:31.881779 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:31.881609 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fhpc7-7qkbj" event={"ID":"f41d65c3-acc5-4de4-a46f-18787ecf7925","Type":"ContainerDied","Data":"3e6efcc7d598345d40a2780ccb402856cdc59f432a8d5687afe48eb632f4e688"} Nov 01 06:08:31.881779 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:31.881739 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e6efcc7d598345d40a2780ccb402856cdc59f432a8d5687afe48eb632f4e688" Nov 01 06:08:39.976114 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:08:39.976072 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:08:42.053038 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:42.053001 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a050afb9-dtvhj"] Nov 01 06:08:42.105817 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:42.105774 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a050afb9-dtvhj"] Nov 01 06:08:43.515284 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:43.515245 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17821c70-5bbf-493f-8d6c-75a217a4e4fb" path="/var/lib/kubelet/pods/17821c70-5bbf-493f-8d6c-75a217a4e4fb/volumes" Nov 01 06:08:45.053232 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:45.053193 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbf06b98-nq2pc"] Nov 01 06:08:45.105221 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:45.105189 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbf06b98-nq2pc"] Nov 01 06:08:45.515854 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:08:45.515816 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2ed195c-edee-4b54-a492-e08e45d2aa10" path="/var/lib/kubelet/pods/f2ed195c-edee-4b54-a492-e08e45d2aa10/volumes" Nov 01 06:09:01.665519 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:01.665471 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-784b1d4f-sjr6m"] Nov 01 06:09:01.666026 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:01.666012 5040 cpu_manager.go:414] "containerMap: removing container" podUID="3e58e9d0-1122-480f-b5ed-60cadb389f23" containerName="revoke-keys" Nov 01 06:09:01.666071 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:01.666026 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e58e9d0-1122-480f-b5ed-60cadb389f23" containerName="revoke-keys" Nov 01 06:09:01.666071 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:01.666048 5040 cpu_manager.go:414] "containerMap: removing container" podUID="f41d65c3-acc5-4de4-a46f-18787ecf7925" containerName="insights-gathering" Nov 01 06:09:01.666071 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:01.666053 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f41d65c3-acc5-4de4-a46f-18787ecf7925" containerName="insights-gathering" Nov 01 06:09:01.666164 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:01.666135 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e58e9d0-1122-480f-b5ed-60cadb389f23" containerName="revoke-keys" Nov 01 06:09:01.666164 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:01.666145 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="f41d65c3-acc5-4de4-a46f-18787ecf7925" containerName="insights-gathering" Nov 01 06:09:01.670859 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:01.670839 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784b1d4f-sjr6m" Nov 01 06:09:01.708009 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:01.707951 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-784b1d4f-sjr6m\" (UID: \"ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784b1d4f-sjr6m" Nov 01 06:09:01.708245 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:01.708218 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-784b1d4f-sjr6m\" (UID: \"ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784b1d4f-sjr6m" Nov 01 06:09:01.708400 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:01.708313 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2cqd\" (UniqueName: \"kubernetes.io/projected/ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05-kube-api-access-x2cqd\") pod \"maas-api-revoke-sub-784b1d4f-sjr6m\" (UID: \"ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784b1d4f-sjr6m" Nov 01 06:09:01.727665 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:01.727609 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-784b1d4f-sjr6m"] Nov 01 06:09:01.772648 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:01.772619 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:09:01.772648 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:01.772655 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:09:01.772648 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:01.772662 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:09:01.772922 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:01.772709 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:09:01.809384 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:01.809336 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-784b1d4f-sjr6m\" (UID: \"ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784b1d4f-sjr6m" Nov 01 06:09:01.809384 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:01.809388 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2cqd\" (UniqueName: \"kubernetes.io/projected/ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05-kube-api-access-x2cqd\") pod \"maas-api-revoke-sub-784b1d4f-sjr6m\" (UID: \"ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784b1d4f-sjr6m" Nov 01 06:09:01.809665 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:01.809507 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-784b1d4f-sjr6m\" (UID: \"ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784b1d4f-sjr6m" Nov 01 06:09:01.810263 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:01.810237 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-784b1d4f-sjr6m\" (UID: \"ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784b1d4f-sjr6m" Nov 01 06:09:01.811835 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:01.811815 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-784b1d4f-sjr6m\" (UID: \"ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784b1d4f-sjr6m" Nov 01 06:09:01.871255 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:01.871215 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2cqd\" (UniqueName: \"kubernetes.io/projected/ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05-kube-api-access-x2cqd\") pod \"maas-api-revoke-sub-784b1d4f-sjr6m\" (UID: \"ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784b1d4f-sjr6m" Nov 01 06:09:01.985151 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:01.985068 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784b1d4f-sjr6m" Nov 01 06:09:02.365780 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:09:02.365734 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffacf7ee_5961_41f5_91aa_2ce1bb6e8c05.slice/crio-bf145bb4cdc60be141e123ea1610d4041637d57eceb7b3e976f99917cb94fafd WatchSource:0}: Error finding container bf145bb4cdc60be141e123ea1610d4041637d57eceb7b3e976f99917cb94fafd: Status 404 returned error can't find the container with id bf145bb4cdc60be141e123ea1610d4041637d57eceb7b3e976f99917cb94fafd Nov 01 06:09:02.366386 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:02.366353 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-784b1d4f-sjr6m"] Nov 01 06:09:03.153950 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:03.153915 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784b1d4f-sjr6m" event={"ID":"ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05","Type":"ContainerStarted","Data":"fa450ce4c63c03a45a7e2ab4f0f7c173a2c3a4bacf483052e8aff329ba46e1a7"} Nov 01 06:09:03.154499 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:03.153965 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784b1d4f-sjr6m" event={"ID":"ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05","Type":"ContainerStarted","Data":"bf145bb4cdc60be141e123ea1610d4041637d57eceb7b3e976f99917cb94fafd"} Nov 01 06:09:03.275992 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:03.275935 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784b1d4f-sjr6m" podStartSLOduration=2.275918073 podStartE2EDuration="2.275918073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:09:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:03.271753954 +0000 UTC m=+1421.926871123" watchObservedRunningTime="2026-11-01 06:09:03.275918073 +0000 UTC m=+1421.931035239" Nov 01 06:09:04.162162 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:04.162126 5040 generic.go:356] "Generic (PLEG): container finished" podID="ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05" containerID="fa450ce4c63c03a45a7e2ab4f0f7c173a2c3a4bacf483052e8aff329ba46e1a7" exitCode=0 Nov 01 06:09:04.162572 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:04.162177 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784b1d4f-sjr6m" event={"ID":"ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05","Type":"ContainerDied","Data":"fa450ce4c63c03a45a7e2ab4f0f7c173a2c3a4bacf483052e8aff329ba46e1a7"} Nov 01 06:09:05.327252 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:05.327221 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784b1d4f-sjr6m" Nov 01 06:09:05.353198 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:05.353164 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05-maas-api-xks-ca\") pod \"ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05\" (UID: \"ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05\") " Nov 01 06:09:05.353386 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:05.353245 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05-maas-api-openshift-ca\") pod \"ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05\" (UID: \"ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05\") " Nov 01 06:09:05.353386 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:05.353266 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05-kube-api-access-x2cqd\" (UniqueName: \"kubernetes.io/projected/ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05-kube-api-access-x2cqd\") pod \"ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05\" (UID: \"ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05\") " Nov 01 06:09:05.353676 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:05.353647 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05-maas-api-openshift-ca" pod "ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05" (UID: "ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:09:05.355427 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:05.355396 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05-maas-api-xks-ca" pod "ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05" (UID: "ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:09:05.355427 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:05.355417 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05-kube-api-access-x2cqd" pod "ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05" (UID: "ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05"). InnerVolumeSpecName "kube-api-access-x2cqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:09:05.455043 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:05.454960 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:09:05.455043 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:05.454988 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:09:05.455043 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:05.454999 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2cqd\" (UniqueName: \"kubernetes.io/projected/ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05-kube-api-access-x2cqd\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:09:06.181689 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:06.181657 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784b1d4f-sjr6m" Nov 01 06:09:06.181689 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:06.181676 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784b1d4f-sjr6m" event={"ID":"ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05","Type":"ContainerDied","Data":"bf145bb4cdc60be141e123ea1610d4041637d57eceb7b3e976f99917cb94fafd"} Nov 01 06:09:06.181906 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:06.181730 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf145bb4cdc60be141e123ea1610d4041637d57eceb7b3e976f99917cb94fafd" Nov 01 06:09:09.051023 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.050983 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl"] Nov 01 06:09:09.051530 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.051515 5040 cpu_manager.go:414] "containerMap: removing container" podUID="ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05" containerName="revoke-keys" Nov 01 06:09:09.051587 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.051531 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05" containerName="revoke-keys" Nov 01 06:09:09.051660 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.051651 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05" containerName="revoke-keys" Nov 01 06:09:09.055251 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.055234 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" Nov 01 06:09:09.095006 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.094968 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f892105-215a-45be-88eb-ac5f03c50c6f-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-kw9tl\" (UID: \"8f892105-215a-45be-88eb-ac5f03c50c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" Nov 01 06:09:09.095203 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.095037 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f892105-215a-45be-88eb-ac5f03c50c6f-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-kw9tl\" (UID: \"8f892105-215a-45be-88eb-ac5f03c50c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" Nov 01 06:09:09.095203 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.095113 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f892105-215a-45be-88eb-ac5f03c50c6f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-kw9tl\" (UID: \"8f892105-215a-45be-88eb-ac5f03c50c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" Nov 01 06:09:09.095203 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.095189 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f892105-215a-45be-88eb-ac5f03c50c6f-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-kw9tl\" (UID: \"8f892105-215a-45be-88eb-ac5f03c50c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" Nov 01 06:09:09.095351 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.095238 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f892105-215a-45be-88eb-ac5f03c50c6f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-kw9tl\" (UID: \"8f892105-215a-45be-88eb-ac5f03c50c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" Nov 01 06:09:09.095387 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.095364 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f892105-215a-45be-88eb-ac5f03c50c6f-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-kw9tl\" (UID: \"8f892105-215a-45be-88eb-ac5f03c50c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" Nov 01 06:09:09.095475 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.095459 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2vqb\" (UniqueName: \"kubernetes.io/projected/8f892105-215a-45be-88eb-ac5f03c50c6f-kube-api-access-b2vqb\") pod \"maas-default-gateway-openshift-default-855f9574cf-kw9tl\" (UID: \"8f892105-215a-45be-88eb-ac5f03c50c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" Nov 01 06:09:09.095591 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.095567 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f892105-215a-45be-88eb-ac5f03c50c6f-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-kw9tl\" (UID: \"8f892105-215a-45be-88eb-ac5f03c50c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" Nov 01 06:09:09.095641 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.095610 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f892105-215a-45be-88eb-ac5f03c50c6f-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-kw9tl\" (UID: \"8f892105-215a-45be-88eb-ac5f03c50c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" Nov 01 06:09:09.101223 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.101198 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44"] Nov 01 06:09:09.105175 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.105160 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" Nov 01 06:09:09.119084 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.119058 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-vqwlh\"" Nov 01 06:09:09.159678 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.159642 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl"] Nov 01 06:09:09.196891 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.196855 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f892105-215a-45be-88eb-ac5f03c50c6f-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-kw9tl\" (UID: \"8f892105-215a-45be-88eb-ac5f03c50c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" Nov 01 06:09:09.197059 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.196900 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-tmf44\" (UID: \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" Nov 01 06:09:09.197059 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.196934 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f892105-215a-45be-88eb-ac5f03c50c6f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-kw9tl\" (UID: \"8f892105-215a-45be-88eb-ac5f03c50c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" Nov 01 06:09:09.197059 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.196970 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-tmf44\" (UID: \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" Nov 01 06:09:09.197059 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.197002 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f892105-215a-45be-88eb-ac5f03c50c6f-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-kw9tl\" (UID: \"8f892105-215a-45be-88eb-ac5f03c50c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" Nov 01 06:09:09.197274 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.197112 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-tmf44\" (UID: \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" Nov 01 06:09:09.197274 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.197153 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-tmf44\" (UID: \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" Nov 01 06:09:09.197274 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.197232 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f892105-215a-45be-88eb-ac5f03c50c6f-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-kw9tl\" (UID: \"8f892105-215a-45be-88eb-ac5f03c50c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" Nov 01 06:09:09.197274 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.197258 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-tmf44\" (UID: \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" Nov 01 06:09:09.197521 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.197328 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2vqb\" (UniqueName: \"kubernetes.io/projected/8f892105-215a-45be-88eb-ac5f03c50c6f-kube-api-access-b2vqb\") pod \"maas-default-gateway-openshift-default-855f9574cf-kw9tl\" (UID: \"8f892105-215a-45be-88eb-ac5f03c50c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" Nov 01 06:09:09.197521 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.197367 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-tmf44\" (UID: \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" Nov 01 06:09:09.197521 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.197418 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f892105-215a-45be-88eb-ac5f03c50c6f-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-kw9tl\" (UID: \"8f892105-215a-45be-88eb-ac5f03c50c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" Nov 01 06:09:09.197521 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.197449 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f892105-215a-45be-88eb-ac5f03c50c6f-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-kw9tl\" (UID: \"8f892105-215a-45be-88eb-ac5f03c50c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" Nov 01 06:09:09.197732 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.197518 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f892105-215a-45be-88eb-ac5f03c50c6f-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-kw9tl\" (UID: \"8f892105-215a-45be-88eb-ac5f03c50c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" Nov 01 06:09:09.197732 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.197550 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-tmf44\" (UID: \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" Nov 01 06:09:09.197732 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.197598 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f892105-215a-45be-88eb-ac5f03c50c6f-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-kw9tl\" (UID: \"8f892105-215a-45be-88eb-ac5f03c50c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" Nov 01 06:09:09.197732 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.197625 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f892105-215a-45be-88eb-ac5f03c50c6f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-kw9tl\" (UID: \"8f892105-215a-45be-88eb-ac5f03c50c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" Nov 01 06:09:09.197732 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.197663 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zznt\" (UniqueName: \"kubernetes.io/projected/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-kube-api-access-6zznt\") pod \"maas-default-gateway-openshift-default-855f9574cf-tmf44\" (UID: \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" Nov 01 06:09:09.198000 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.197922 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-tmf44\" (UID: \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" Nov 01 06:09:09.198000 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.197944 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f892105-215a-45be-88eb-ac5f03c50c6f-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-kw9tl\" (UID: \"8f892105-215a-45be-88eb-ac5f03c50c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" Nov 01 06:09:09.198107 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.198008 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f892105-215a-45be-88eb-ac5f03c50c6f-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-kw9tl\" (UID: \"8f892105-215a-45be-88eb-ac5f03c50c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" Nov 01 06:09:09.198231 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.198208 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f892105-215a-45be-88eb-ac5f03c50c6f-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-kw9tl\" (UID: \"8f892105-215a-45be-88eb-ac5f03c50c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" Nov 01 06:09:09.198498 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.198456 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f892105-215a-45be-88eb-ac5f03c50c6f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-kw9tl\" (UID: \"8f892105-215a-45be-88eb-ac5f03c50c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" Nov 01 06:09:09.199887 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.199860 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f892105-215a-45be-88eb-ac5f03c50c6f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-kw9tl\" (UID: \"8f892105-215a-45be-88eb-ac5f03c50c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" Nov 01 06:09:09.200051 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.200034 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f892105-215a-45be-88eb-ac5f03c50c6f-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-kw9tl\" (UID: \"8f892105-215a-45be-88eb-ac5f03c50c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" Nov 01 06:09:09.224771 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.224735 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44"] Nov 01 06:09:09.259129 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.259093 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f892105-215a-45be-88eb-ac5f03c50c6f-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-kw9tl\" (UID: \"8f892105-215a-45be-88eb-ac5f03c50c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" Nov 01 06:09:09.259284 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.259146 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2vqb\" (UniqueName: \"kubernetes.io/projected/8f892105-215a-45be-88eb-ac5f03c50c6f-kube-api-access-b2vqb\") pod \"maas-default-gateway-openshift-default-855f9574cf-kw9tl\" (UID: \"8f892105-215a-45be-88eb-ac5f03c50c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" Nov 01 06:09:09.298757 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.298723 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-tmf44\" (UID: \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" Nov 01 06:09:09.299012 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.298783 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-tmf44\" (UID: \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" Nov 01 06:09:09.299012 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.298803 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-tmf44\" (UID: \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" Nov 01 06:09:09.299012 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.298846 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-tmf44\" (UID: \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" Nov 01 06:09:09.299012 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.298884 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-tmf44\" (UID: \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" Nov 01 06:09:09.299012 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.298939 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-tmf44\" (UID: \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" Nov 01 06:09:09.299012 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.298995 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zznt\" (UniqueName: \"kubernetes.io/projected/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-kube-api-access-6zznt\") pod \"maas-default-gateway-openshift-default-855f9574cf-tmf44\" (UID: \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" Nov 01 06:09:09.299301 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.299031 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-tmf44\" (UID: \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" Nov 01 06:09:09.299301 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.299095 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-tmf44\" (UID: \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" Nov 01 06:09:09.299371 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.299326 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-tmf44\" (UID: \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" Nov 01 06:09:09.299473 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.299450 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-tmf44\" (UID: \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" Nov 01 06:09:09.299696 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.299672 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-tmf44\" (UID: \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" Nov 01 06:09:09.299787 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.299732 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-tmf44\" (UID: \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" Nov 01 06:09:09.299991 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.299972 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-tmf44\" (UID: \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" Nov 01 06:09:09.301819 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.301753 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-tmf44\" (UID: \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" Nov 01 06:09:09.301989 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.301971 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-tmf44\" (UID: \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" Nov 01 06:09:09.360596 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.360556 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-tmf44\" (UID: \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" Nov 01 06:09:09.360732 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.360703 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zznt\" (UniqueName: \"kubernetes.io/projected/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-kube-api-access-6zznt\") pod \"maas-default-gateway-openshift-default-855f9574cf-tmf44\" (UID: \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" Nov 01 06:09:09.369584 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.369559 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" Nov 01 06:09:09.418829 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.418798 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" Nov 01 06:09:09.805900 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.805861 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl"] Nov 01 06:09:09.840878 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:09:09.840850 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f892105_215a_45be_88eb_ac5f03c50c6f.slice/crio-afb2c597c6993658ed70cf93be5e5f9e3b0c91d90667e8b5d392caf40f73c340 WatchSource:0}: Error finding container afb2c597c6993658ed70cf93be5e5f9e3b0c91d90667e8b5d392caf40f73c340: Status 404 returned error can't find the container with id afb2c597c6993658ed70cf93be5e5f9e3b0c91d90667e8b5d392caf40f73c340 Nov 01 06:09:09.843251 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.843217 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:09:09.843340 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.843306 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:09:09.843340 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.843336 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:09:09.858177 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:09:09.858149 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ea5cbab_4aa1_431f_a4a4_64a2f324e3cb.slice/crio-0d51ad36cec04a8369ed22d7243670c07e7522ac3f421cabe53ad17e4795c5dc WatchSource:0}: Error finding container 0d51ad36cec04a8369ed22d7243670c07e7522ac3f421cabe53ad17e4795c5dc: Status 404 returned error can't find the container with id 0d51ad36cec04a8369ed22d7243670c07e7522ac3f421cabe53ad17e4795c5dc Nov 01 06:09:09.858922 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.858897 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44"] Nov 01 06:09:09.860737 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.860706 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:09:09.860785 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.860769 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:09:09.860822 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:09.860797 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:09:09.992507 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:09:09.990250 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:09:10.222132 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:10.221965 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" event={"ID":"8f892105-215a-45be-88eb-ac5f03c50c6f","Type":"ContainerStarted","Data":"8d9b161c59493586ca1b206ecd47fb4d23a6ab568bbf2e36e78720af3225727e"} Nov 01 06:09:10.222132 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:10.222031 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" event={"ID":"8f892105-215a-45be-88eb-ac5f03c50c6f","Type":"ContainerStarted","Data":"afb2c597c6993658ed70cf93be5e5f9e3b0c91d90667e8b5d392caf40f73c340"} Nov 01 06:09:10.224790 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:10.224754 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" event={"ID":"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb","Type":"ContainerStarted","Data":"b9d9a0e02f39fa4413b97efdb62c664305162a959ce91cce1390f9b0adcf1108"} Nov 01 06:09:10.224946 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:10.224814 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" event={"ID":"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb","Type":"ContainerStarted","Data":"0d51ad36cec04a8369ed22d7243670c07e7522ac3f421cabe53ad17e4795c5dc"} Nov 01 06:09:10.343752 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:10.343698 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" podStartSLOduration=1.343684012 podStartE2EDuration="1.343684012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:10.342826696 +0000 UTC m=+1428.997943868" watchObservedRunningTime="2026-11-01 06:09:10.343684012 +0000 UTC m=+1428.998801173" Nov 01 06:09:10.370358 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:10.370320 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" Nov 01 06:09:10.371637 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:10.371609 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-kw9tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 01 06:09:10.371774 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:10.371669 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" podUID="8f892105-215a-45be-88eb-ac5f03c50c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 01 06:09:10.419682 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:10.419649 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" Nov 01 06:09:10.421100 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:10.421071 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tmf44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 01 06:09:10.421255 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:10.421117 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" podUID="6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 01 06:09:10.466513 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:10.466446 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" podStartSLOduration=1.466432355 podStartE2EDuration="1.466432355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:10.462824592 +0000 UTC m=+1429.117941759" watchObservedRunningTime="2026-11-01 06:09:10.466432355 +0000 UTC m=+1429.121549588" Nov 01 06:09:11.370850 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:11.370818 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-kw9tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 01 06:09:11.371644 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:11.370870 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" podUID="8f892105-215a-45be-88eb-ac5f03c50c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 01 06:09:11.425499 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:11.423118 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tmf44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 01 06:09:11.425499 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:11.423177 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" podUID="6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 01 06:09:12.054618 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:12.054571 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fc2a0b8-649s8"] Nov 01 06:09:12.058980 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:12.058952 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fc2a0b8-649s8"] Nov 01 06:09:12.370112 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:12.370030 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-kw9tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 01 06:09:12.370112 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:12.370079 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" podUID="8f892105-215a-45be-88eb-ac5f03c50c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 01 06:09:12.437503 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:12.436584 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tmf44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 01 06:09:12.437503 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:12.436646 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" podUID="6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 01 06:09:13.370064 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:13.370026 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-kw9tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 01 06:09:13.370263 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:13.370079 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" podUID="8f892105-215a-45be-88eb-ac5f03c50c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 01 06:09:13.421506 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:13.421190 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tmf44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 01 06:09:13.421506 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:13.421256 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" podUID="6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 01 06:09:13.550253 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:13.550218 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8" path="/var/lib/kubelet/pods/9dee132c-5bc0-4b5d-95dc-eabfe5b1a2d8/volumes" Nov 01 06:09:14.370006 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:14.369960 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-kw9tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 01 06:09:14.370233 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:14.370048 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" podUID="8f892105-215a-45be-88eb-ac5f03c50c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 01 06:09:14.427504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:14.426914 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tmf44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 01 06:09:14.427504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:14.426996 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" podUID="6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 01 06:09:15.370296 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:15.370256 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-kw9tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 01 06:09:15.370746 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:15.370329 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" podUID="8f892105-215a-45be-88eb-ac5f03c50c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 01 06:09:15.422504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:15.421892 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tmf44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 01 06:09:15.422504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:15.421957 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" podUID="6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 01 06:09:16.370361 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:16.370329 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-kw9tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 01 06:09:16.370875 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:16.370846 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" podUID="8f892105-215a-45be-88eb-ac5f03c50c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 01 06:09:16.426504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:16.426103 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tmf44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 01 06:09:16.426504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:16.426169 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" podUID="6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 01 06:09:17.370866 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:17.370821 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-kw9tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 01 06:09:17.371267 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:17.370890 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" podUID="8f892105-215a-45be-88eb-ac5f03c50c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 01 06:09:17.419384 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:17.419343 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tmf44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 01 06:09:17.419604 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:17.419422 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" podUID="6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 01 06:09:18.052299 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:18.052258 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8c5ce51-cgh5j"] Nov 01 06:09:18.100026 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:18.099992 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8c5ce51-cgh5j"] Nov 01 06:09:18.370258 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:18.370153 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-kw9tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 01 06:09:18.370258 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:18.370223 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" podUID="8f892105-215a-45be-88eb-ac5f03c50c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 01 06:09:19.063419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:19.063384 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" Nov 01 06:09:19.063967 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:19.063947 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" Nov 01 06:09:19.065203 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:19.065178 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" Nov 01 06:09:19.387703 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:19.386858 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" Nov 01 06:09:19.390217 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:19.390187 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" Nov 01 06:09:19.391418 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:19.391399 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" Nov 01 06:09:19.517429 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:19.517395 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97686a71-a832-4014-83de-9c9c710f1b62" path="/var/lib/kubelet/pods/97686a71-a832-4014-83de-9c9c710f1b62/volumes" Nov 01 06:09:21.752919 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:21.752891 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-58d523-5887dbc76d-7fs98_feb6d45c-3a57-4a1e-863c-8c0d7972d453/payload-processing/0.log" Nov 01 06:09:21.810252 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:21.810212 5040 scope.go:122] "RemoveContainer" containerID="02038dfa70f16ef0855d455cfd5d3a85a2dfa62284139c2cc7846cb58d2cf9dc" Nov 01 06:09:21.820695 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:21.820669 5040 scope.go:122] "RemoveContainer" containerID="ed2514b62a6d09cc013c8a0f6bdd2cf11da9ea47eb508cfae77160143163d1dc" Nov 01 06:09:21.830282 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:21.830261 5040 scope.go:122] "RemoveContainer" containerID="a84e8aabdca1f19d664eebaabb569e921f2f97d61011f7bba2ad549f4071b295" Nov 01 06:09:21.839790 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:21.839770 5040 scope.go:122] "RemoveContainer" containerID="96bf31c5422b4ac116d768e29eb07f2213558a2461be29ad870d9a04312e9e47" Nov 01 06:09:21.848800 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:21.848778 5040 scope.go:122] "RemoveContainer" containerID="dde2bd13c2bcf06fc881f6ae79920de0db3e0ce90f89b77b65c0dda2aa014907" Nov 01 06:09:21.858033 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:21.858013 5040 scope.go:122] "RemoveContainer" containerID="c573b02578890d4b1897e484b6dc64c118365d7af77faebd65a60d76933ba09e" Nov 01 06:09:22.475679 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:22.475651 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-2d3f8d-7d966b6c54-v7flb_07af255e-cc07-4e90-bdba-828ac197fa53/payload-processing/0.log" Nov 01 06:09:22.690092 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:22.690064 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-2d3f8d-7d966b6c54-v7flb_07af255e-cc07-4e90-bdba-828ac197fa53/payload-processing/0.log" Nov 01 06:09:27.132256 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:27.132204 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-695f5e6f-5886d"] Nov 01 06:09:27.178050 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:27.178014 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-695f5e6f-5886d"] Nov 01 06:09:27.521844 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:27.521804 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7a48e16-9f97-4c4d-888e-9e5434f7233d" path="/var/lib/kubelet/pods/a7a48e16-9f97-4c4d-888e-9e5434f7233d/volumes" Nov 01 06:09:39.972389 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:09:39.972343 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:09:54.064050 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:54.064012 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl"] Nov 01 06:09:54.068325 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:54.068305 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" Nov 01 06:09:54.202259 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:54.202222 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0ce9454-b676-4e7d-abd3-34d76aa2c787-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-brrdl\" (UID: \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" Nov 01 06:09:54.202259 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:54.202264 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0ce9454-b676-4e7d-abd3-34d76aa2c787-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-brrdl\" (UID: \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" Nov 01 06:09:54.202516 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:54.202286 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln9vr\" (UniqueName: \"kubernetes.io/projected/b0ce9454-b676-4e7d-abd3-34d76aa2c787-kube-api-access-ln9vr\") pod \"maas-default-gateway-openshift-default-855f9574cf-brrdl\" (UID: \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" Nov 01 06:09:54.202516 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:54.202399 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0ce9454-b676-4e7d-abd3-34d76aa2c787-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-brrdl\" (UID: \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" Nov 01 06:09:54.202516 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:54.202448 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ce9454-b676-4e7d-abd3-34d76aa2c787-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-brrdl\" (UID: \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" Nov 01 06:09:54.202636 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:54.202532 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0ce9454-b676-4e7d-abd3-34d76aa2c787-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-brrdl\" (UID: \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" Nov 01 06:09:54.202636 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:54.202565 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0ce9454-b676-4e7d-abd3-34d76aa2c787-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-brrdl\" (UID: \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" Nov 01 06:09:54.202636 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:54.202597 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0ce9454-b676-4e7d-abd3-34d76aa2c787-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-brrdl\" (UID: \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" Nov 01 06:09:54.202636 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:54.202612 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ce9454-b676-4e7d-abd3-34d76aa2c787-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-brrdl\" (UID: \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" Nov 01 06:09:54.282712 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:54.282677 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl"] Nov 01 06:09:54.303149 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:54.303115 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0ce9454-b676-4e7d-abd3-34d76aa2c787-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-brrdl\" (UID: \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" Nov 01 06:09:54.303338 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:54.303174 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0ce9454-b676-4e7d-abd3-34d76aa2c787-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-brrdl\" (UID: \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" Nov 01 06:09:54.303338 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:54.303225 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0ce9454-b676-4e7d-abd3-34d76aa2c787-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-brrdl\" (UID: \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" Nov 01 06:09:54.303338 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:54.303247 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ce9454-b676-4e7d-abd3-34d76aa2c787-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-brrdl\" (UID: \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" Nov 01 06:09:54.303338 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:54.303288 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0ce9454-b676-4e7d-abd3-34d76aa2c787-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-brrdl\" (UID: \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" Nov 01 06:09:54.303338 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:54.303321 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0ce9454-b676-4e7d-abd3-34d76aa2c787-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-brrdl\" (UID: \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" Nov 01 06:09:54.303663 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:54.303428 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln9vr\" (UniqueName: \"kubernetes.io/projected/b0ce9454-b676-4e7d-abd3-34d76aa2c787-kube-api-access-ln9vr\") pod \"maas-default-gateway-openshift-default-855f9574cf-brrdl\" (UID: \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" Nov 01 06:09:54.303663 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:54.303581 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0ce9454-b676-4e7d-abd3-34d76aa2c787-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-brrdl\" (UID: \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" Nov 01 06:09:54.303816 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:54.303666 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ce9454-b676-4e7d-abd3-34d76aa2c787-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-brrdl\" (UID: \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" Nov 01 06:09:54.303936 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:54.303915 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0ce9454-b676-4e7d-abd3-34d76aa2c787-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-brrdl\" (UID: \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" Nov 01 06:09:54.304004 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:54.303989 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ce9454-b676-4e7d-abd3-34d76aa2c787-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-brrdl\" (UID: \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" Nov 01 06:09:54.304053 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:54.304000 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0ce9454-b676-4e7d-abd3-34d76aa2c787-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-brrdl\" (UID: \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" Nov 01 06:09:54.304112 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:54.304003 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0ce9454-b676-4e7d-abd3-34d76aa2c787-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-brrdl\" (UID: \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" Nov 01 06:09:54.304295 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:54.304276 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ce9454-b676-4e7d-abd3-34d76aa2c787-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-brrdl\" (UID: \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" Nov 01 06:09:54.305723 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:54.305702 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0ce9454-b676-4e7d-abd3-34d76aa2c787-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-brrdl\" (UID: \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" Nov 01 06:09:54.305953 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:54.305933 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0ce9454-b676-4e7d-abd3-34d76aa2c787-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-brrdl\" (UID: \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" Nov 01 06:09:54.365707 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:54.365637 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0ce9454-b676-4e7d-abd3-34d76aa2c787-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-brrdl\" (UID: \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" Nov 01 06:09:54.365853 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:54.365788 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln9vr\" (UniqueName: \"kubernetes.io/projected/b0ce9454-b676-4e7d-abd3-34d76aa2c787-kube-api-access-ln9vr\") pod \"maas-default-gateway-openshift-default-855f9574cf-brrdl\" (UID: \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" Nov 01 06:09:54.382075 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:54.382049 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" Nov 01 06:09:54.820456 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:09:54.820424 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ce9454_b676_4e7d_abd3_34d76aa2c787.slice/crio-1441df6a726d78d3f626d4f47384efdec36ede31741937f048ecee4038691f88 WatchSource:0}: Error finding container 1441df6a726d78d3f626d4f47384efdec36ede31741937f048ecee4038691f88: Status 404 returned error can't find the container with id 1441df6a726d78d3f626d4f47384efdec36ede31741937f048ecee4038691f88 Nov 01 06:09:54.820978 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:54.820957 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl"] Nov 01 06:09:54.822842 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:54.822812 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:09:54.822900 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:54.822876 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:09:54.822936 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:54.822908 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:09:55.730765 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:55.730722 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" event={"ID":"b0ce9454-b676-4e7d-abd3-34d76aa2c787","Type":"ContainerStarted","Data":"de1372a895791e68004c9e81f9d24fd2773533ae800a18e317124abe98f8caff"} Nov 01 06:09:55.731257 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:55.730790 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" event={"ID":"b0ce9454-b676-4e7d-abd3-34d76aa2c787","Type":"ContainerStarted","Data":"1441df6a726d78d3f626d4f47384efdec36ede31741937f048ecee4038691f88"} Nov 01 06:09:55.911503 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:55.910261 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" podStartSLOduration=1.9102381899999998 podStartE2EDuration="1.91023819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:55.90805845 +0000 UTC m=+1474.563175624" watchObservedRunningTime="2026-11-01 06:09:55.91023819 +0000 UTC m=+1474.565355357" Nov 01 06:09:56.382824 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:56.382791 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" Nov 01 06:09:56.384232 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:56.384205 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-brrdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 01 06:09:56.384402 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:56.384263 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" podUID="b0ce9454-b676-4e7d-abd3-34d76aa2c787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 01 06:09:57.383091 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:57.383056 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-brrdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 01 06:09:57.383629 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:57.383119 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" podUID="b0ce9454-b676-4e7d-abd3-34d76aa2c787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 01 06:09:58.382855 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:58.382820 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-brrdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 01 06:09:58.383037 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:58.382885 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" podUID="b0ce9454-b676-4e7d-abd3-34d76aa2c787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 01 06:09:59.383102 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:59.383068 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-brrdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 01 06:09:59.383741 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:09:59.383126 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" podUID="b0ce9454-b676-4e7d-abd3-34d76aa2c787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 01 06:10:00.382492 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:00.382448 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-brrdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 01 06:10:00.382682 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:00.382519 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" podUID="b0ce9454-b676-4e7d-abd3-34d76aa2c787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 01 06:10:01.382584 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:01.382542 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-brrdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 01 06:10:01.383084 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:01.382622 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" podUID="b0ce9454-b676-4e7d-abd3-34d76aa2c787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 01 06:10:02.383375 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:02.383337 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-brrdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 01 06:10:02.383892 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:02.383411 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" podUID="b0ce9454-b676-4e7d-abd3-34d76aa2c787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 01 06:10:03.387012 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:03.386976 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" Nov 01 06:10:03.387511 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:03.387467 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" Nov 01 06:10:03.388610 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:03.388591 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" Nov 01 06:10:06.115264 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:06.115220 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ead5c4a-fnffv"] Nov 01 06:10:06.125511 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:06.125419 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ead5c4a-fnffv" Nov 01 06:10:06.156522 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:06.156495 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e993ed84-a606-4616-b122-3319a82c5633-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ead5c4a-fnffv\" (UID: \"e993ed84-a606-4616-b122-3319a82c5633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ead5c4a-fnffv" Nov 01 06:10:06.156650 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:06.156570 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e993ed84-a606-4616-b122-3319a82c5633-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ead5c4a-fnffv\" (UID: \"e993ed84-a606-4616-b122-3319a82c5633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ead5c4a-fnffv" Nov 01 06:10:06.156690 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:06.156661 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qpsc\" (UniqueName: \"kubernetes.io/projected/e993ed84-a606-4616-b122-3319a82c5633-kube-api-access-8qpsc\") pod \"maas-api-revoke-sub-9ead5c4a-fnffv\" (UID: \"e993ed84-a606-4616-b122-3319a82c5633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ead5c4a-fnffv" Nov 01 06:10:06.176276 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:06.176248 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ead5c4a-fnffv"] Nov 01 06:10:06.178840 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:06.178815 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:10:06.221320 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:06.221294 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:10:06.221320 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:06.221293 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:10:06.221545 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:06.221326 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:10:06.258193 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:06.258161 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e993ed84-a606-4616-b122-3319a82c5633-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ead5c4a-fnffv\" (UID: \"e993ed84-a606-4616-b122-3319a82c5633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ead5c4a-fnffv" Nov 01 06:10:06.258361 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:06.258263 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e993ed84-a606-4616-b122-3319a82c5633-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ead5c4a-fnffv\" (UID: \"e993ed84-a606-4616-b122-3319a82c5633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ead5c4a-fnffv" Nov 01 06:10:06.258361 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:06.258312 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qpsc\" (UniqueName: \"kubernetes.io/projected/e993ed84-a606-4616-b122-3319a82c5633-kube-api-access-8qpsc\") pod \"maas-api-revoke-sub-9ead5c4a-fnffv\" (UID: \"e993ed84-a606-4616-b122-3319a82c5633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ead5c4a-fnffv" Nov 01 06:10:06.258985 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:06.258957 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e993ed84-a606-4616-b122-3319a82c5633-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ead5c4a-fnffv\" (UID: \"e993ed84-a606-4616-b122-3319a82c5633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ead5c4a-fnffv" Nov 01 06:10:06.260457 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:06.260441 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e993ed84-a606-4616-b122-3319a82c5633-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ead5c4a-fnffv\" (UID: \"e993ed84-a606-4616-b122-3319a82c5633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ead5c4a-fnffv" Nov 01 06:10:06.319819 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:06.319794 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qpsc\" (UniqueName: \"kubernetes.io/projected/e993ed84-a606-4616-b122-3319a82c5633-kube-api-access-8qpsc\") pod \"maas-api-revoke-sub-9ead5c4a-fnffv\" (UID: \"e993ed84-a606-4616-b122-3319a82c5633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ead5c4a-fnffv" Nov 01 06:10:06.440502 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:06.440410 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ead5c4a-fnffv" Nov 01 06:10:06.825514 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:06.819282 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ead5c4a-fnffv"] Nov 01 06:10:07.838387 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:07.838353 5040 generic.go:356] "Generic (PLEG): container finished" podID="e993ed84-a606-4616-b122-3319a82c5633" containerID="c4e51e91787b5727c54ff2ad477f26a82570c94acddd357612b2d4fa82aa859d" exitCode=0 Nov 01 06:10:07.838864 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:07.838443 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ead5c4a-fnffv" event={"ID":"e993ed84-a606-4616-b122-3319a82c5633","Type":"ContainerDied","Data":"c4e51e91787b5727c54ff2ad477f26a82570c94acddd357612b2d4fa82aa859d"} Nov 01 06:10:07.838864 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:07.838548 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ead5c4a-fnffv" event={"ID":"e993ed84-a606-4616-b122-3319a82c5633","Type":"ContainerStarted","Data":"5e71e0f2ddfca8032d60002a87cba399da37a9224001949b9afc4276bcaaefb5"} Nov 01 06:10:09.008031 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:09.008003 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ead5c4a-fnffv" Nov 01 06:10:09.092779 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:09.092748 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e993ed84-a606-4616-b122-3319a82c5633-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e993ed84-a606-4616-b122-3319a82c5633-maas-api-xks-ca\") pod \"e993ed84-a606-4616-b122-3319a82c5633\" (UID: \"e993ed84-a606-4616-b122-3319a82c5633\") " Nov 01 06:10:09.092779 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:09.092786 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e993ed84-a606-4616-b122-3319a82c5633-kube-api-access-8qpsc\" (UniqueName: \"kubernetes.io/projected/e993ed84-a606-4616-b122-3319a82c5633-kube-api-access-8qpsc\") pod \"e993ed84-a606-4616-b122-3319a82c5633\" (UID: \"e993ed84-a606-4616-b122-3319a82c5633\") " Nov 01 06:10:09.092981 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:09.092845 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e993ed84-a606-4616-b122-3319a82c5633-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e993ed84-a606-4616-b122-3319a82c5633-maas-api-openshift-ca\") pod \"e993ed84-a606-4616-b122-3319a82c5633\" (UID: \"e993ed84-a606-4616-b122-3319a82c5633\") " Nov 01 06:10:09.093282 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:09.093249 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e993ed84-a606-4616-b122-3319a82c5633-maas-api-openshift-ca" pod "e993ed84-a606-4616-b122-3319a82c5633" (UID: "e993ed84-a606-4616-b122-3319a82c5633"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:10:09.093379 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:09.093364 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e993ed84-a606-4616-b122-3319a82c5633-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:10:09.094890 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:09.094863 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e993ed84-a606-4616-b122-3319a82c5633-maas-api-xks-ca" pod "e993ed84-a606-4616-b122-3319a82c5633" (UID: "e993ed84-a606-4616-b122-3319a82c5633"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:10:09.094995 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:09.094971 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e993ed84-a606-4616-b122-3319a82c5633-kube-api-access-8qpsc" pod "e993ed84-a606-4616-b122-3319a82c5633" (UID: "e993ed84-a606-4616-b122-3319a82c5633"). InnerVolumeSpecName "kube-api-access-8qpsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:10:09.194351 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:09.194308 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e993ed84-a606-4616-b122-3319a82c5633-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:10:09.194351 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:09.194347 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qpsc\" (UniqueName: \"kubernetes.io/projected/e993ed84-a606-4616-b122-3319a82c5633-kube-api-access-8qpsc\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:10:09.854615 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:09.854527 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ead5c4a-fnffv" event={"ID":"e993ed84-a606-4616-b122-3319a82c5633","Type":"ContainerDied","Data":"5e71e0f2ddfca8032d60002a87cba399da37a9224001949b9afc4276bcaaefb5"} Nov 01 06:10:09.854615 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:09.854592 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e71e0f2ddfca8032d60002a87cba399da37a9224001949b9afc4276bcaaefb5" Nov 01 06:10:09.854615 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:09.854534 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ead5c4a-fnffv" Nov 01 06:10:09.973259 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:10:09.973221 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:10:10.048665 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:10.048627 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0a7600e-cls4g"] Nov 01 06:10:10.052914 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:10.052888 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0a7600e-cls4g"] Nov 01 06:10:11.516071 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:11.516027 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31442774-8721-46d1-a4af-3fe60a925167" path="/var/lib/kubelet/pods/31442774-8721-46d1-a4af-3fe60a925167/volumes" Nov 01 06:10:21.736157 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:21.736124 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 06:10:21.739724 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:21.739700 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 06:10:21.912973 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:21.912944 5040 scope.go:122] "RemoveContainer" containerID="3b795a245719fb7a3f1b0c39db0f36700924890eba354dd5ca7ab4ad5fe264f0" Nov 01 06:10:21.925515 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:21.925389 5040 scope.go:122] "RemoveContainer" containerID="a36800499c5c3d8299d90485b1215eb529fc8090255ad4aff048c85b92c0cee3" Nov 01 06:10:21.942590 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:21.942562 5040 scope.go:122] "RemoveContainer" containerID="ee0c00b6834f25692dda0e5ea3d0f2e4cd60170135cda8bdbb50c1abd596b96d" Nov 01 06:10:21.969627 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:21.969599 5040 scope.go:122] "RemoveContainer" containerID="3b064ba62af995d265c7d18099d34e1d7ffa18a1f26f1ef833222080326da7a4" Nov 01 06:10:21.983132 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:21.983107 5040 scope.go:122] "RemoveContainer" containerID="e18ebde0483a0923a8df922da4ceca6f6d8f0d59c099a5c2c6e50f7dc69a3fb3" Nov 01 06:10:25.049602 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:25.049568 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-637fa5fc-5g4z9"] Nov 01 06:10:25.099389 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:25.099358 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-637fa5fc-5g4z9"] Nov 01 06:10:25.515952 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:25.515919 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e1dd328-b0f8-4ea0-9aab-93837559f559" path="/var/lib/kubelet/pods/3e1dd328-b0f8-4ea0-9aab-93837559f559/volumes" Nov 01 06:10:39.973500 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:10:39.973447 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:10:52.048864 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:52.048831 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e729141-x95rn"] Nov 01 06:10:52.099098 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:52.099062 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e729141-x95rn"] Nov 01 06:10:53.516122 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:53.516090 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cb7101b-6199-46dc-a4b0-a7832ec19011" path="/var/lib/kubelet/pods/2cb7101b-6199-46dc-a4b0-a7832ec19011/volumes" Nov 01 06:10:54.310551 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:54.310509 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58d523-d4bn2"] Nov 01 06:10:54.311038 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:54.311025 5040 cpu_manager.go:414] "containerMap: removing container" podUID="e993ed84-a606-4616-b122-3319a82c5633" containerName="revoke-keys" Nov 01 06:10:54.311083 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:54.311039 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e993ed84-a606-4616-b122-3319a82c5633" containerName="revoke-keys" Nov 01 06:10:54.311161 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:54.311151 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="e993ed84-a606-4616-b122-3319a82c5633" containerName="revoke-keys" Nov 01 06:10:54.315740 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:54.315721 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58d523-d4bn2" Nov 01 06:10:54.372066 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:54.372024 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58d523-d4bn2"] Nov 01 06:10:54.377589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:54.377558 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b8a332-acb6-4075-a630-cc2f89ea7048-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-58d523-d4bn2\" (UID: \"a8b8a332-acb6-4075-a630-cc2f89ea7048\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58d523-d4bn2" Nov 01 06:10:54.377762 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:54.377600 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk6hc\" (UniqueName: \"kubernetes.io/projected/a8b8a332-acb6-4075-a630-cc2f89ea7048-kube-api-access-zk6hc\") pod \"maas-api-revoke-keys-e2e-ipp-a-58d523-d4bn2\" (UID: \"a8b8a332-acb6-4075-a630-cc2f89ea7048\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58d523-d4bn2" Nov 01 06:10:54.377762 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:54.377696 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b8a332-acb6-4075-a630-cc2f89ea7048-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-58d523-d4bn2\" (UID: \"a8b8a332-acb6-4075-a630-cc2f89ea7048\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58d523-d4bn2" Nov 01 06:10:54.417547 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:54.417516 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:10:54.417547 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:54.417545 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:10:54.417724 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:54.417545 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:10:54.419027 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:54.419010 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:10:54.478684 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:54.478661 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b8a332-acb6-4075-a630-cc2f89ea7048-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-58d523-d4bn2\" (UID: \"a8b8a332-acb6-4075-a630-cc2f89ea7048\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58d523-d4bn2" Nov 01 06:10:54.478806 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:54.478693 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk6hc\" (UniqueName: \"kubernetes.io/projected/a8b8a332-acb6-4075-a630-cc2f89ea7048-kube-api-access-zk6hc\") pod \"maas-api-revoke-keys-e2e-ipp-a-58d523-d4bn2\" (UID: \"a8b8a332-acb6-4075-a630-cc2f89ea7048\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58d523-d4bn2" Nov 01 06:10:54.478861 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:54.478818 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b8a332-acb6-4075-a630-cc2f89ea7048-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-58d523-d4bn2\" (UID: \"a8b8a332-acb6-4075-a630-cc2f89ea7048\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58d523-d4bn2" Nov 01 06:10:54.479292 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:54.479270 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b8a332-acb6-4075-a630-cc2f89ea7048-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-58d523-d4bn2\" (UID: \"a8b8a332-acb6-4075-a630-cc2f89ea7048\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58d523-d4bn2" Nov 01 06:10:54.481141 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:54.481114 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b8a332-acb6-4075-a630-cc2f89ea7048-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-58d523-d4bn2\" (UID: \"a8b8a332-acb6-4075-a630-cc2f89ea7048\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58d523-d4bn2" Nov 01 06:10:54.540307 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:54.540281 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk6hc\" (UniqueName: \"kubernetes.io/projected/a8b8a332-acb6-4075-a630-cc2f89ea7048-kube-api-access-zk6hc\") pod \"maas-api-revoke-keys-e2e-ipp-a-58d523-d4bn2\" (UID: \"a8b8a332-acb6-4075-a630-cc2f89ea7048\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58d523-d4bn2" Nov 01 06:10:54.629464 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:54.629385 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58d523-d4bn2" Nov 01 06:10:55.013512 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:55.013464 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58d523-d4bn2"] Nov 01 06:10:55.013912 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:10:55.013885 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8b8a332_acb6_4075_a630_cc2f89ea7048.slice/crio-2525e866b690e27c60e7ea72a754a7f03b475d053bd47b764d1d9466436b5c0f WatchSource:0}: Error finding container 2525e866b690e27c60e7ea72a754a7f03b475d053bd47b764d1d9466436b5c0f: Status 404 returned error can't find the container with id 2525e866b690e27c60e7ea72a754a7f03b475d053bd47b764d1d9466436b5c0f Nov 01 06:10:55.223792 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:55.223758 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58d523-d4bn2" event={"ID":"a8b8a332-acb6-4075-a630-cc2f89ea7048","Type":"ContainerStarted","Data":"2525e866b690e27c60e7ea72a754a7f03b475d053bd47b764d1d9466436b5c0f"} Nov 01 06:10:56.232346 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:56.232259 5040 generic.go:356] "Generic (PLEG): container finished" podID="a8b8a332-acb6-4075-a630-cc2f89ea7048" containerID="4639b7a60a2bef9706e99765375aaf28c20dec80f94a484151470526c38ccb7b" exitCode=0 Nov 01 06:10:56.232346 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:56.232324 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58d523-d4bn2" event={"ID":"a8b8a332-acb6-4075-a630-cc2f89ea7048","Type":"ContainerDied","Data":"4639b7a60a2bef9706e99765375aaf28c20dec80f94a484151470526c38ccb7b"} Nov 01 06:10:57.392892 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:57.392867 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58d523-d4bn2" Nov 01 06:10:57.412682 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:57.412656 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8b8a332-acb6-4075-a630-cc2f89ea7048-kube-api-access-zk6hc\" (UniqueName: \"kubernetes.io/projected/a8b8a332-acb6-4075-a630-cc2f89ea7048-kube-api-access-zk6hc\") pod \"a8b8a332-acb6-4075-a630-cc2f89ea7048\" (UID: \"a8b8a332-acb6-4075-a630-cc2f89ea7048\") " Nov 01 06:10:57.412798 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:57.412750 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8b8a332-acb6-4075-a630-cc2f89ea7048-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b8a332-acb6-4075-a630-cc2f89ea7048-maas-api-xks-ca\") pod \"a8b8a332-acb6-4075-a630-cc2f89ea7048\" (UID: \"a8b8a332-acb6-4075-a630-cc2f89ea7048\") " Nov 01 06:10:57.412846 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:57.412830 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8b8a332-acb6-4075-a630-cc2f89ea7048-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b8a332-acb6-4075-a630-cc2f89ea7048-maas-api-openshift-ca\") pod \"a8b8a332-acb6-4075-a630-cc2f89ea7048\" (UID: \"a8b8a332-acb6-4075-a630-cc2f89ea7048\") " Nov 01 06:10:57.413159 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:57.413135 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8b8a332-acb6-4075-a630-cc2f89ea7048-maas-api-openshift-ca" pod "a8b8a332-acb6-4075-a630-cc2f89ea7048" (UID: "a8b8a332-acb6-4075-a630-cc2f89ea7048"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:10:57.414730 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:57.414707 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8b8a332-acb6-4075-a630-cc2f89ea7048-kube-api-access-zk6hc" pod "a8b8a332-acb6-4075-a630-cc2f89ea7048" (UID: "a8b8a332-acb6-4075-a630-cc2f89ea7048"). InnerVolumeSpecName "kube-api-access-zk6hc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:10:57.414819 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:57.414765 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8b8a332-acb6-4075-a630-cc2f89ea7048-maas-api-xks-ca" pod "a8b8a332-acb6-4075-a630-cc2f89ea7048" (UID: "a8b8a332-acb6-4075-a630-cc2f89ea7048"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:10:57.514132 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:57.514100 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b8a332-acb6-4075-a630-cc2f89ea7048-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:10:57.514132 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:57.514135 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b8a332-acb6-4075-a630-cc2f89ea7048-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:10:57.514132 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:57.514150 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk6hc\" (UniqueName: \"kubernetes.io/projected/a8b8a332-acb6-4075-a630-cc2f89ea7048-kube-api-access-zk6hc\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:10:58.249947 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:58.249911 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58d523-d4bn2" event={"ID":"a8b8a332-acb6-4075-a630-cc2f89ea7048","Type":"ContainerDied","Data":"2525e866b690e27c60e7ea72a754a7f03b475d053bd47b764d1d9466436b5c0f"} Nov 01 06:10:58.250119 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:58.249964 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2525e866b690e27c60e7ea72a754a7f03b475d053bd47b764d1d9466436b5c0f" Nov 01 06:10:58.250119 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:10:58.249919 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58d523-d4bn2" Nov 01 06:11:02.049314 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:02.049277 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cb9816e-b97hn"] Nov 01 06:11:02.099627 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:02.099594 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cb9816e-b97hn"] Nov 01 06:11:02.850623 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:02.850589 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h"] Nov 01 06:11:03.048142 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:03.048108 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c18e3e98-j984d"] Nov 01 06:11:03.053610 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:03.053579 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c18e3e98-j984d"] Nov 01 06:11:03.289726 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:03.289655 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" podUID="0073efde-c913-424b-93e8-bda25a24a209" containerName="istio-proxy" containerID="cri-o://a43b59ddb22ed0bec2898be848b183d8bbb19b5ebf26279a1087e91bbd9b435e" gracePeriod=30 Nov 01 06:11:03.515578 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:03.515544 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="629b293f-5a09-48c0-b154-85d8c81ca510" path="/var/lib/kubelet/pods/629b293f-5a09-48c0-b154-85d8c81ca510/volumes" Nov 01 06:11:03.516096 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:03.516073 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8892735-cb03-43a6-8312-808304b332a3" path="/var/lib/kubelet/pods/f8892735-cb03-43a6-8312-808304b332a3/volumes" Nov 01 06:11:07.174401 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:07.174366 5040 patch_prober.go:29] interesting pod/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 06:11:07.174830 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:07.174422 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" podUID="0073efde-c913-424b-93e8-bda25a24a209" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 06:11:08.966736 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:08.966705 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:11:09.051234 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.051200 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0073efde-c913-424b-93e8-bda25a24a209-kube-api-access-6xc7c\" (UniqueName: \"kubernetes.io/projected/0073efde-c913-424b-93e8-bda25a24a209-kube-api-access-6xc7c\") pod \"0073efde-c913-424b-93e8-bda25a24a209\" (UID: \"0073efde-c913-424b-93e8-bda25a24a209\") " Nov 01 06:11:09.051404 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.051280 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0073efde-c913-424b-93e8-bda25a24a209-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0073efde-c913-424b-93e8-bda25a24a209-workload-certs\") pod \"0073efde-c913-424b-93e8-bda25a24a209\" (UID: \"0073efde-c913-424b-93e8-bda25a24a209\") " Nov 01 06:11:09.051404 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.051313 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0073efde-c913-424b-93e8-bda25a24a209-istio-token\" (UniqueName: \"kubernetes.io/projected/0073efde-c913-424b-93e8-bda25a24a209-istio-token\") pod \"0073efde-c913-424b-93e8-bda25a24a209\" (UID: \"0073efde-c913-424b-93e8-bda25a24a209\") " Nov 01 06:11:09.051527 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.051413 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0073efde-c913-424b-93e8-bda25a24a209-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0073efde-c913-424b-93e8-bda25a24a209-istiod-ca-cert\") pod \"0073efde-c913-424b-93e8-bda25a24a209\" (UID: \"0073efde-c913-424b-93e8-bda25a24a209\") " Nov 01 06:11:09.051597 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.051553 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0073efde-c913-424b-93e8-bda25a24a209-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0073efde-c913-424b-93e8-bda25a24a209-istio-data\") pod \"0073efde-c913-424b-93e8-bda25a24a209\" (UID: \"0073efde-c913-424b-93e8-bda25a24a209\") " Nov 01 06:11:09.051656 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.051595 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0073efde-c913-424b-93e8-bda25a24a209-workload-certs" pod "0073efde-c913-424b-93e8-bda25a24a209" (UID: "0073efde-c913-424b-93e8-bda25a24a209"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:11:09.051656 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.051601 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0073efde-c913-424b-93e8-bda25a24a209-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0073efde-c913-424b-93e8-bda25a24a209-istio-envoy\") pod \"0073efde-c913-424b-93e8-bda25a24a209\" (UID: \"0073efde-c913-424b-93e8-bda25a24a209\") " Nov 01 06:11:09.051762 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.051680 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0073efde-c913-424b-93e8-bda25a24a209-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0073efde-c913-424b-93e8-bda25a24a209-istio-podinfo\") pod \"0073efde-c913-424b-93e8-bda25a24a209\" (UID: \"0073efde-c913-424b-93e8-bda25a24a209\") " Nov 01 06:11:09.051841 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.051817 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0073efde-c913-424b-93e8-bda25a24a209-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0073efde-c913-424b-93e8-bda25a24a209-credential-socket\") pod \"0073efde-c913-424b-93e8-bda25a24a209\" (UID: \"0073efde-c913-424b-93e8-bda25a24a209\") " Nov 01 06:11:09.051904 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.051843 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0073efde-c913-424b-93e8-bda25a24a209-istio-data" pod "0073efde-c913-424b-93e8-bda25a24a209" (UID: "0073efde-c913-424b-93e8-bda25a24a209"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:11:09.051904 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.051854 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0073efde-c913-424b-93e8-bda25a24a209-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0073efde-c913-424b-93e8-bda25a24a209-workload-socket\") pod \"0073efde-c913-424b-93e8-bda25a24a209\" (UID: \"0073efde-c913-424b-93e8-bda25a24a209\") " Nov 01 06:11:09.052011 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.051896 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0073efde-c913-424b-93e8-bda25a24a209-istiod-ca-cert" pod "0073efde-c913-424b-93e8-bda25a24a209" (UID: "0073efde-c913-424b-93e8-bda25a24a209"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:11:09.052065 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.052051 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0073efde-c913-424b-93e8-bda25a24a209-workload-socket" pod "0073efde-c913-424b-93e8-bda25a24a209" (UID: "0073efde-c913-424b-93e8-bda25a24a209"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:11:09.052210 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.052185 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0073efde-c913-424b-93e8-bda25a24a209-credential-socket" pod "0073efde-c913-424b-93e8-bda25a24a209" (UID: "0073efde-c913-424b-93e8-bda25a24a209"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:11:09.052432 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.052414 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0073efde-c913-424b-93e8-bda25a24a209-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:11:09.052529 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.052437 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0073efde-c913-424b-93e8-bda25a24a209-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:11:09.052529 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.052451 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0073efde-c913-424b-93e8-bda25a24a209-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:11:09.052529 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.052465 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0073efde-c913-424b-93e8-bda25a24a209-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:11:09.052529 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.052506 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0073efde-c913-424b-93e8-bda25a24a209-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:11:09.053713 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.053684 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0073efde-c913-424b-93e8-bda25a24a209-kube-api-access-6xc7c" pod "0073efde-c913-424b-93e8-bda25a24a209" (UID: "0073efde-c913-424b-93e8-bda25a24a209"). InnerVolumeSpecName "kube-api-access-6xc7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:11:09.053820 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.053725 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0073efde-c913-424b-93e8-bda25a24a209-istio-token" pod "0073efde-c913-424b-93e8-bda25a24a209" (UID: "0073efde-c913-424b-93e8-bda25a24a209"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:11:09.054008 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.053987 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0073efde-c913-424b-93e8-bda25a24a209-istio-envoy" pod "0073efde-c913-424b-93e8-bda25a24a209" (UID: "0073efde-c913-424b-93e8-bda25a24a209"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:11:09.054067 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.053984 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0073efde-c913-424b-93e8-bda25a24a209-istio-podinfo" pod "0073efde-c913-424b-93e8-bda25a24a209" (UID: "0073efde-c913-424b-93e8-bda25a24a209"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:11:09.153387 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.153300 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xc7c\" (UniqueName: \"kubernetes.io/projected/0073efde-c913-424b-93e8-bda25a24a209-kube-api-access-6xc7c\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:11:09.153387 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.153337 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0073efde-c913-424b-93e8-bda25a24a209-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:11:09.153387 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.153351 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0073efde-c913-424b-93e8-bda25a24a209-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:11:09.153387 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.153363 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0073efde-c913-424b-93e8-bda25a24a209-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:11:09.359198 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.359165 5040 generic.go:356] "Generic (PLEG): container finished" podID="0073efde-c913-424b-93e8-bda25a24a209" containerID="a43b59ddb22ed0bec2898be848b183d8bbb19b5ebf26279a1087e91bbd9b435e" exitCode=0 Nov 01 06:11:09.359407 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.359215 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" Nov 01 06:11:09.359407 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.359224 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" event={"ID":"0073efde-c913-424b-93e8-bda25a24a209","Type":"ContainerDied","Data":"a43b59ddb22ed0bec2898be848b183d8bbb19b5ebf26279a1087e91bbd9b435e"} Nov 01 06:11:09.359407 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.359292 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h" event={"ID":"0073efde-c913-424b-93e8-bda25a24a209","Type":"ContainerDied","Data":"9e0bd4ff4980a8b7851082fec60572a5e7537752b41b56a0f2448fe24d4b487f"} Nov 01 06:11:09.359407 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.359322 5040 scope.go:122] "RemoveContainer" containerID="a43b59ddb22ed0bec2898be848b183d8bbb19b5ebf26279a1087e91bbd9b435e" Nov 01 06:11:09.370632 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.370616 5040 scope.go:122] "RemoveContainer" containerID="73a7065d2e4f82ade11bd5eba20661c60ba5233484bfa20430f66a700499a859" Nov 01 06:11:09.380673 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.380634 5040 scope.go:122] "RemoveContainer" containerID="a43b59ddb22ed0bec2898be848b183d8bbb19b5ebf26279a1087e91bbd9b435e" Nov 01 06:11:09.380940 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:11:09.380920 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a43b59ddb22ed0bec2898be848b183d8bbb19b5ebf26279a1087e91bbd9b435e\": container with ID starting with a43b59ddb22ed0bec2898be848b183d8bbb19b5ebf26279a1087e91bbd9b435e not found: ID does not exist" containerID="a43b59ddb22ed0bec2898be848b183d8bbb19b5ebf26279a1087e91bbd9b435e" Nov 01 06:11:09.381033 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.380953 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a43b59ddb22ed0bec2898be848b183d8bbb19b5ebf26279a1087e91bbd9b435e"} err="failed to get container status \"a43b59ddb22ed0bec2898be848b183d8bbb19b5ebf26279a1087e91bbd9b435e\": rpc error: code = NotFound desc = could not find container \"a43b59ddb22ed0bec2898be848b183d8bbb19b5ebf26279a1087e91bbd9b435e\": container with ID starting with a43b59ddb22ed0bec2898be848b183d8bbb19b5ebf26279a1087e91bbd9b435e not found: ID does not exist" Nov 01 06:11:09.381033 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.380979 5040 scope.go:122] "RemoveContainer" containerID="73a7065d2e4f82ade11bd5eba20661c60ba5233484bfa20430f66a700499a859" Nov 01 06:11:09.381232 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:11:09.381209 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73a7065d2e4f82ade11bd5eba20661c60ba5233484bfa20430f66a700499a859\": container with ID starting with 73a7065d2e4f82ade11bd5eba20661c60ba5233484bfa20430f66a700499a859 not found: ID does not exist" containerID="73a7065d2e4f82ade11bd5eba20661c60ba5233484bfa20430f66a700499a859" Nov 01 06:11:09.381314 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.381238 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73a7065d2e4f82ade11bd5eba20661c60ba5233484bfa20430f66a700499a859"} err="failed to get container status \"73a7065d2e4f82ade11bd5eba20661c60ba5233484bfa20430f66a700499a859\": rpc error: code = NotFound desc = could not find container \"73a7065d2e4f82ade11bd5eba20661c60ba5233484bfa20430f66a700499a859\": container with ID starting with 73a7065d2e4f82ade11bd5eba20661c60ba5233484bfa20430f66a700499a859 not found: ID does not exist" Nov 01 06:11:09.532206 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.532173 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h"] Nov 01 06:11:09.542583 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:09.542551 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-bd29h"] Nov 01 06:11:09.977540 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:11:09.977437 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:11:10.159746 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:10.159712 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-58d523-5887dbc76d-7fs98"] Nov 01 06:11:10.159933 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:10.159913 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-58d523-5887dbc76d-7fs98" podUID="feb6d45c-3a57-4a1e-863c-8c0d7972d453" containerName="payload-processing" containerID="cri-o://20407a10f1597e26ec22f3ab05ab31043b436bca555362dc37e5133883ccd838" gracePeriod=30 Nov 01 06:11:10.213072 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:10.213043 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7d69f54555-8ltqs"] Nov 01 06:11:10.213254 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:10.213233 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7d69f54555-8ltqs" podUID="7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd" containerName="payload-pre-processing" containerID="cri-o://a95a49ad3c960df2691928169bdd2714803ace0f70dbd14528313282d6c0b350" gracePeriod=30 Nov 01 06:11:10.372073 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:10.372043 5040 generic.go:356] "Generic (PLEG): container finished" podID="7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd" containerID="a95a49ad3c960df2691928169bdd2714803ace0f70dbd14528313282d6c0b350" exitCode=0 Nov 01 06:11:10.372274 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:10.372112 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7d69f54555-8ltqs" event={"ID":"7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd","Type":"ContainerDied","Data":"a95a49ad3c960df2691928169bdd2714803ace0f70dbd14528313282d6c0b350"} Nov 01 06:11:10.375531 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:10.375500 5040 generic.go:356] "Generic (PLEG): container finished" podID="feb6d45c-3a57-4a1e-863c-8c0d7972d453" containerID="20407a10f1597e26ec22f3ab05ab31043b436bca555362dc37e5133883ccd838" exitCode=0 Nov 01 06:11:10.375660 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:10.375547 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-58d523-5887dbc76d-7fs98" event={"ID":"feb6d45c-3a57-4a1e-863c-8c0d7972d453","Type":"ContainerDied","Data":"20407a10f1597e26ec22f3ab05ab31043b436bca555362dc37e5133883ccd838"} Nov 01 06:11:10.501111 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:10.500771 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-58d523-5887dbc76d-7fs98" Nov 01 06:11:10.541672 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:10.541647 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7d69f54555-8ltqs" Nov 01 06:11:10.571897 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:10.571865 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feb6d45c-3a57-4a1e-863c-8c0d7972d453-kube-api-access-s86nl\" (UniqueName: \"kubernetes.io/projected/feb6d45c-3a57-4a1e-863c-8c0d7972d453-kube-api-access-s86nl\") pod \"feb6d45c-3a57-4a1e-863c-8c0d7972d453\" (UID: \"feb6d45c-3a57-4a1e-863c-8c0d7972d453\") " Nov 01 06:11:10.572046 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:10.571961 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feb6d45c-3a57-4a1e-863c-8c0d7972d453-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feb6d45c-3a57-4a1e-863c-8c0d7972d453-plugins-config-volume\") pod \"feb6d45c-3a57-4a1e-863c-8c0d7972d453\" (UID: \"feb6d45c-3a57-4a1e-863c-8c0d7972d453\") " Nov 01 06:11:10.572292 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:10.572270 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feb6d45c-3a57-4a1e-863c-8c0d7972d453-plugins-config-volume" pod "feb6d45c-3a57-4a1e-863c-8c0d7972d453" (UID: "feb6d45c-3a57-4a1e-863c-8c0d7972d453"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:11:10.573939 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:10.573919 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feb6d45c-3a57-4a1e-863c-8c0d7972d453-kube-api-access-s86nl" pod "feb6d45c-3a57-4a1e-863c-8c0d7972d453" (UID: "feb6d45c-3a57-4a1e-863c-8c0d7972d453"). InnerVolumeSpecName "kube-api-access-s86nl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:11:10.735776 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:10.735549 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd-plugins-config-volume\") pod \"7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd\" (UID: \"7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd\") " Nov 01 06:11:10.735776 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:10.735708 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd-kube-api-access-kplxx\" (UniqueName: \"kubernetes.io/projected/7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd-kube-api-access-kplxx\") pod \"7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd\" (UID: \"7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd\") " Nov 01 06:11:10.737802 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:10.736567 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s86nl\" (UniqueName: \"kubernetes.io/projected/feb6d45c-3a57-4a1e-863c-8c0d7972d453-kube-api-access-s86nl\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:11:10.737802 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:10.736590 5040 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feb6d45c-3a57-4a1e-863c-8c0d7972d453-plugins-config-volume\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:11:10.765116 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:10.750391 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd-kube-api-access-kplxx" pod "7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd" (UID: "7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd"). InnerVolumeSpecName "kube-api-access-kplxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:11:10.765116 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:10.755549 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd-plugins-config-volume" pod "7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd" (UID: "7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:11:10.837556 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:10.837517 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kplxx\" (UniqueName: \"kubernetes.io/projected/7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd-kube-api-access-kplxx\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:11:10.837556 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:10.837557 5040 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd-plugins-config-volume\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:11:11.384688 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:11.384657 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7d69f54555-8ltqs" Nov 01 06:11:11.385192 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:11.384650 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7d69f54555-8ltqs" event={"ID":"7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd","Type":"ContainerDied","Data":"0b11fa5dbd3770570495856a17881180b127b8a906030eef84f938e53cbb747b"} Nov 01 06:11:11.385192 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:11.384844 5040 scope.go:122] "RemoveContainer" containerID="a95a49ad3c960df2691928169bdd2714803ace0f70dbd14528313282d6c0b350" Nov 01 06:11:11.386462 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:11.386436 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-58d523-5887dbc76d-7fs98" event={"ID":"feb6d45c-3a57-4a1e-863c-8c0d7972d453","Type":"ContainerDied","Data":"ea81b57ca6199e3fc9c7ea0990a05e1bc693a5f0ec35cff97158345556fa79b1"} Nov 01 06:11:11.386652 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:11.386563 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-58d523-5887dbc76d-7fs98" Nov 01 06:11:11.396023 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:11.396002 5040 scope.go:122] "RemoveContainer" containerID="20407a10f1597e26ec22f3ab05ab31043b436bca555362dc37e5133883ccd838" Nov 01 06:11:11.517128 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:11.517098 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0073efde-c913-424b-93e8-bda25a24a209" path="/var/lib/kubelet/pods/0073efde-c913-424b-93e8-bda25a24a209/volumes" Nov 01 06:11:11.563286 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:11.563252 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7d69f54555-8ltqs"] Nov 01 06:11:11.589833 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:11.589796 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7d69f54555-8ltqs"] Nov 01 06:11:11.676046 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:11.676012 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-58d523-5887dbc76d-7fs98"] Nov 01 06:11:11.682721 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:11.682684 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-58d523-5887dbc76d-7fs98"] Nov 01 06:11:11.794663 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:11.794582 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58d523-d4bn2"] Nov 01 06:11:11.799460 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:11.799433 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58d523-d4bn2"] Nov 01 06:11:12.013568 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:12.013535 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf"] Nov 01 06:11:12.013814 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:12.013788 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" podUID="a325a624-c22a-4425-9002-ad68a3c22314" containerName="istio-proxy" containerID="cri-o://d9afdaaba3e9450be83ed80c1eb9d49491201ab0fabb20dabfd7468e206510b3" gracePeriod=30 Nov 01 06:11:13.370722 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:13.370683 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/llmisvc-controller-manager-6466c554d8-788tv" podUID="72a66fbc-a5fe-42f7-8362-fe8e2a62025f" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.49:8081/readyz\": dial tcp 10.132.0.49:8081: connect: connection refused" Nov 01 06:11:13.405195 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:13.405162 5040 generic.go:356] "Generic (PLEG): container finished" podID="72a66fbc-a5fe-42f7-8362-fe8e2a62025f" containerID="2452f40fc1b747b3d606b15ed5d6a59633332ea278dae2b65e7965f94388a8f5" exitCode=1 Nov 01 06:11:13.405351 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:13.405207 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-788tv" event={"ID":"72a66fbc-a5fe-42f7-8362-fe8e2a62025f","Type":"ContainerDied","Data":"2452f40fc1b747b3d606b15ed5d6a59633332ea278dae2b65e7965f94388a8f5"} Nov 01 06:11:13.405582 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:13.405568 5040 scope.go:122] "RemoveContainer" containerID="2452f40fc1b747b3d606b15ed5d6a59633332ea278dae2b65e7965f94388a8f5" Nov 01 06:11:13.405981 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:13.405968 5040 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 06:11:13.514582 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:13.514553 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd" path="/var/lib/kubelet/pods/7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd/volumes" Nov 01 06:11:13.515023 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:13.515008 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8b8a332-acb6-4075-a630-cc2f89ea7048" path="/var/lib/kubelet/pods/a8b8a332-acb6-4075-a630-cc2f89ea7048/volumes" Nov 01 06:11:13.515375 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:13.515363 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feb6d45c-3a57-4a1e-863c-8c0d7972d453" path="/var/lib/kubelet/pods/feb6d45c-3a57-4a1e-863c-8c0d7972d453/volumes" Nov 01 06:11:14.414782 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:14.414746 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-788tv" event={"ID":"72a66fbc-a5fe-42f7-8362-fe8e2a62025f","Type":"ContainerStarted","Data":"a353ec1da49d944643acebc1dc25abcf8e1bfaa5cb38757ae681f5f8dd399911"} Nov 01 06:11:14.415282 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:14.415043 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-788tv" Nov 01 06:11:16.048489 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:16.048451 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ffc4cba-nd6dz"] Nov 01 06:11:16.098699 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:16.098660 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ffc4cba-nd6dz"] Nov 01 06:11:17.296673 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.296604 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:11:17.311532 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.311506 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a325a624-c22a-4425-9002-ad68a3c22314-istio-token\" (UniqueName: \"kubernetes.io/projected/a325a624-c22a-4425-9002-ad68a3c22314-istio-token\") pod \"a325a624-c22a-4425-9002-ad68a3c22314\" (UID: \"a325a624-c22a-4425-9002-ad68a3c22314\") " Nov 01 06:11:17.311707 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.311554 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a325a624-c22a-4425-9002-ad68a3c22314-kube-api-access-94ltf\" (UniqueName: \"kubernetes.io/projected/a325a624-c22a-4425-9002-ad68a3c22314-kube-api-access-94ltf\") pod \"a325a624-c22a-4425-9002-ad68a3c22314\" (UID: \"a325a624-c22a-4425-9002-ad68a3c22314\") " Nov 01 06:11:17.311707 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.311596 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a325a624-c22a-4425-9002-ad68a3c22314-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a325a624-c22a-4425-9002-ad68a3c22314-istiod-ca-cert\") pod \"a325a624-c22a-4425-9002-ad68a3c22314\" (UID: \"a325a624-c22a-4425-9002-ad68a3c22314\") " Nov 01 06:11:17.311707 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.311649 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a325a624-c22a-4425-9002-ad68a3c22314-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a325a624-c22a-4425-9002-ad68a3c22314-istio-podinfo\") pod \"a325a624-c22a-4425-9002-ad68a3c22314\" (UID: \"a325a624-c22a-4425-9002-ad68a3c22314\") " Nov 01 06:11:17.311707 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.311696 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a325a624-c22a-4425-9002-ad68a3c22314-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a325a624-c22a-4425-9002-ad68a3c22314-credential-socket\") pod \"a325a624-c22a-4425-9002-ad68a3c22314\" (UID: \"a325a624-c22a-4425-9002-ad68a3c22314\") " Nov 01 06:11:17.311925 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.311790 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a325a624-c22a-4425-9002-ad68a3c22314-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a325a624-c22a-4425-9002-ad68a3c22314-istio-data\") pod \"a325a624-c22a-4425-9002-ad68a3c22314\" (UID: \"a325a624-c22a-4425-9002-ad68a3c22314\") " Nov 01 06:11:17.311925 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.311817 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a325a624-c22a-4425-9002-ad68a3c22314-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a325a624-c22a-4425-9002-ad68a3c22314-workload-certs\") pod \"a325a624-c22a-4425-9002-ad68a3c22314\" (UID: \"a325a624-c22a-4425-9002-ad68a3c22314\") " Nov 01 06:11:17.311925 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.311874 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a325a624-c22a-4425-9002-ad68a3c22314-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a325a624-c22a-4425-9002-ad68a3c22314-workload-socket\") pod \"a325a624-c22a-4425-9002-ad68a3c22314\" (UID: \"a325a624-c22a-4425-9002-ad68a3c22314\") " Nov 01 06:11:17.311925 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.311905 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a325a624-c22a-4425-9002-ad68a3c22314-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a325a624-c22a-4425-9002-ad68a3c22314-istio-envoy\") pod \"a325a624-c22a-4425-9002-ad68a3c22314\" (UID: \"a325a624-c22a-4425-9002-ad68a3c22314\") " Nov 01 06:11:17.312138 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.312098 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a325a624-c22a-4425-9002-ad68a3c22314-istiod-ca-cert" pod "a325a624-c22a-4425-9002-ad68a3c22314" (UID: "a325a624-c22a-4425-9002-ad68a3c22314"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:11:17.312286 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.312255 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a325a624-c22a-4425-9002-ad68a3c22314-istio-data" pod "a325a624-c22a-4425-9002-ad68a3c22314" (UID: "a325a624-c22a-4425-9002-ad68a3c22314"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:11:17.312524 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.312470 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a325a624-c22a-4425-9002-ad68a3c22314-workload-socket" pod "a325a624-c22a-4425-9002-ad68a3c22314" (UID: "a325a624-c22a-4425-9002-ad68a3c22314"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:11:17.312643 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.312617 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a325a624-c22a-4425-9002-ad68a3c22314-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:11:17.312731 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.312641 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a325a624-c22a-4425-9002-ad68a3c22314-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:11:17.312731 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.312658 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a325a624-c22a-4425-9002-ad68a3c22314-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:11:17.312731 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.312655 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a325a624-c22a-4425-9002-ad68a3c22314-credential-socket" pod "a325a624-c22a-4425-9002-ad68a3c22314" (UID: "a325a624-c22a-4425-9002-ad68a3c22314"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:11:17.313023 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.312989 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a325a624-c22a-4425-9002-ad68a3c22314-workload-certs" pod "a325a624-c22a-4425-9002-ad68a3c22314" (UID: "a325a624-c22a-4425-9002-ad68a3c22314"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:11:17.313996 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.313975 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a325a624-c22a-4425-9002-ad68a3c22314-kube-api-access-94ltf" pod "a325a624-c22a-4425-9002-ad68a3c22314" (UID: "a325a624-c22a-4425-9002-ad68a3c22314"). InnerVolumeSpecName "kube-api-access-94ltf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:11:17.314147 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.314127 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a325a624-c22a-4425-9002-ad68a3c22314-istio-token" pod "a325a624-c22a-4425-9002-ad68a3c22314" (UID: "a325a624-c22a-4425-9002-ad68a3c22314"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:11:17.314228 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.314160 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a325a624-c22a-4425-9002-ad68a3c22314-istio-podinfo" pod "a325a624-c22a-4425-9002-ad68a3c22314" (UID: "a325a624-c22a-4425-9002-ad68a3c22314"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:11:17.314360 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.314341 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a325a624-c22a-4425-9002-ad68a3c22314-istio-envoy" pod "a325a624-c22a-4425-9002-ad68a3c22314" (UID: "a325a624-c22a-4425-9002-ad68a3c22314"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:11:17.413625 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.413594 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a325a624-c22a-4425-9002-ad68a3c22314-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:11:17.413625 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.413620 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a325a624-c22a-4425-9002-ad68a3c22314-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:11:17.413625 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.413631 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94ltf\" (UniqueName: \"kubernetes.io/projected/a325a624-c22a-4425-9002-ad68a3c22314-kube-api-access-94ltf\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:11:17.413852 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.413641 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a325a624-c22a-4425-9002-ad68a3c22314-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:11:17.413852 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.413650 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a325a624-c22a-4425-9002-ad68a3c22314-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:11:17.413852 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.413658 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a325a624-c22a-4425-9002-ad68a3c22314-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:11:17.441336 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.441305 5040 generic.go:356] "Generic (PLEG): container finished" podID="a325a624-c22a-4425-9002-ad68a3c22314" containerID="d9afdaaba3e9450be83ed80c1eb9d49491201ab0fabb20dabfd7468e206510b3" exitCode=0 Nov 01 06:11:17.441336 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.441348 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" event={"ID":"a325a624-c22a-4425-9002-ad68a3c22314","Type":"ContainerDied","Data":"d9afdaaba3e9450be83ed80c1eb9d49491201ab0fabb20dabfd7468e206510b3"} Nov 01 06:11:17.441602 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.441353 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" Nov 01 06:11:17.441602 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.441390 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf" event={"ID":"a325a624-c22a-4425-9002-ad68a3c22314","Type":"ContainerDied","Data":"2ddd1f61ee4516f7a6a55c961388f9349e3b8c523b5cc0e5ecd726d31c77453b"} Nov 01 06:11:17.441602 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.441415 5040 scope.go:122] "RemoveContainer" containerID="d9afdaaba3e9450be83ed80c1eb9d49491201ab0fabb20dabfd7468e206510b3" Nov 01 06:11:17.456647 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.453413 5040 scope.go:122] "RemoveContainer" containerID="c390947f4f51a2b41e5f3cc13b3e0531642cc06e939c569d148ce41319b5775b" Nov 01 06:11:17.464253 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.464222 5040 scope.go:122] "RemoveContainer" containerID="d9afdaaba3e9450be83ed80c1eb9d49491201ab0fabb20dabfd7468e206510b3" Nov 01 06:11:17.464550 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:11:17.464528 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9afdaaba3e9450be83ed80c1eb9d49491201ab0fabb20dabfd7468e206510b3\": container with ID starting with d9afdaaba3e9450be83ed80c1eb9d49491201ab0fabb20dabfd7468e206510b3 not found: ID does not exist" containerID="d9afdaaba3e9450be83ed80c1eb9d49491201ab0fabb20dabfd7468e206510b3" Nov 01 06:11:17.464646 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.464559 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9afdaaba3e9450be83ed80c1eb9d49491201ab0fabb20dabfd7468e206510b3"} err="failed to get container status \"d9afdaaba3e9450be83ed80c1eb9d49491201ab0fabb20dabfd7468e206510b3\": rpc error: code = NotFound desc = could not find container \"d9afdaaba3e9450be83ed80c1eb9d49491201ab0fabb20dabfd7468e206510b3\": container with ID starting with d9afdaaba3e9450be83ed80c1eb9d49491201ab0fabb20dabfd7468e206510b3 not found: ID does not exist" Nov 01 06:11:17.464646 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.464576 5040 scope.go:122] "RemoveContainer" containerID="c390947f4f51a2b41e5f3cc13b3e0531642cc06e939c569d148ce41319b5775b" Nov 01 06:11:17.464825 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:11:17.464806 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c390947f4f51a2b41e5f3cc13b3e0531642cc06e939c569d148ce41319b5775b\": container with ID starting with c390947f4f51a2b41e5f3cc13b3e0531642cc06e939c569d148ce41319b5775b not found: ID does not exist" containerID="c390947f4f51a2b41e5f3cc13b3e0531642cc06e939c569d148ce41319b5775b" Nov 01 06:11:17.464889 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.464830 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c390947f4f51a2b41e5f3cc13b3e0531642cc06e939c569d148ce41319b5775b"} err="failed to get container status \"c390947f4f51a2b41e5f3cc13b3e0531642cc06e939c569d148ce41319b5775b\": rpc error: code = NotFound desc = could not find container \"c390947f4f51a2b41e5f3cc13b3e0531642cc06e939c569d148ce41319b5775b\": container with ID starting with c390947f4f51a2b41e5f3cc13b3e0531642cc06e939c569d148ce41319b5775b not found: ID does not exist" Nov 01 06:11:17.517874 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.517838 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ae36837-6f06-4bfb-87b1-4bcfed69167e" path="/var/lib/kubelet/pods/7ae36837-6f06-4bfb-87b1-4bcfed69167e/volumes" Nov 01 06:11:17.583251 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.583154 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf"] Nov 01 06:11:17.587906 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:17.587868 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-58d523-openshift-default-5b5bc444df-lp2mf"] Nov 01 06:11:19.049253 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:19.049211 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d52e3446-bprl4"] Nov 01 06:11:19.053413 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:19.053381 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d52e3446-bprl4"] Nov 01 06:11:19.057515 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:19.057493 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c0400ea-9wkhc"] Nov 01 06:11:19.099543 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:19.099505 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c0400ea-9wkhc"] Nov 01 06:11:19.515414 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:19.515371 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65dbb0a6-e671-4131-945a-5a15c9433b25" path="/var/lib/kubelet/pods/65dbb0a6-e671-4131-945a-5a15c9433b25/volumes" Nov 01 06:11:19.515940 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:19.515920 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a325a624-c22a-4425-9002-ad68a3c22314" path="/var/lib/kubelet/pods/a325a624-c22a-4425-9002-ad68a3c22314/volumes" Nov 01 06:11:19.516510 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:19.516470 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c10af860-64f5-4bc5-bb86-93b48de1a0f3" path="/var/lib/kubelet/pods/c10af860-64f5-4bc5-bb86-93b48de1a0f3/volumes" Nov 01 06:11:22.058005 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:22.057972 5040 scope.go:122] "RemoveContainer" containerID="475a7c868936f26f22eac7c682b882a932afbd667840098aff1042c7ebe3c368" Nov 01 06:11:22.068515 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:22.068460 5040 scope.go:122] "RemoveContainer" containerID="f3460af015e74a44070445d64b84d0c057f348e5762b4ee0fed9bdd903dd3d70" Nov 01 06:11:22.078864 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:22.078840 5040 scope.go:122] "RemoveContainer" containerID="5ca903bd862220e6f744b5fb89c73db0d6f5addc4d7c765b8dfb8112ab2ef4d6" Nov 01 06:11:30.049029 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:30.048993 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf6b940e-nhpmt"] Nov 01 06:11:30.099352 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:30.099316 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf6b940e-nhpmt"] Nov 01 06:11:31.517025 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:31.516934 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="998ef654-d73e-4a90-9e6a-f45d90cc959c" path="/var/lib/kubelet/pods/998ef654-d73e-4a90-9e6a-f45d90cc959c/volumes" Nov 01 06:11:35.050843 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:35.050813 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46c05f3b-r78db"] Nov 01 06:11:35.100807 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:35.100760 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46c05f3b-r78db"] Nov 01 06:11:35.529660 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:35.529605 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="852e1076-47cc-4f39-bc85-88719ff826a5" path="/var/lib/kubelet/pods/852e1076-47cc-4f39-bc85-88719ff826a5/volumes" Nov 01 06:11:39.983529 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:11:39.983492 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:11:45.425530 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:45.425496 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-788tv" Nov 01 06:11:49.049830 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:49.049787 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7585a10d-cx7cs"] Nov 01 06:11:49.099958 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:49.099918 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7585a10d-cx7cs"] Nov 01 06:11:49.515826 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:49.515794 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b900531c-f6b3-4b8f-901d-91e8535a2062" path="/var/lib/kubelet/pods/b900531c-f6b3-4b8f-901d-91e8535a2062/volumes" Nov 01 06:11:55.046687 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:55.046650 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb56174b-g7p4r"] Nov 01 06:11:55.105167 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:55.105127 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb56174b-g7p4r"] Nov 01 06:11:55.517312 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:11:55.517277 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8d4b89a-a81e-4763-a236-5bc2ad562a7e" path="/var/lib/kubelet/pods/b8d4b89a-a81e-4763-a236-5bc2ad562a7e/volumes" Nov 01 06:12:00.049657 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:00.049613 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c50fe9e4-7278t"] Nov 01 06:12:00.100160 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:00.100121 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c50fe9e4-7278t"] Nov 01 06:12:01.523490 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:01.523440 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06c33506-de91-47b3-9495-fdb20ff7d816" path="/var/lib/kubelet/pods/06c33506-de91-47b3-9495-fdb20ff7d816/volumes" Nov 01 06:12:03.320954 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:03.320917 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26"] Nov 01 06:12:03.321410 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:03.321124 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" podUID="e1442af5-c9a0-4995-be20-1f65aa32d0bd" containerName="istio-proxy" containerID="cri-o://782f05aa12faf7e0e9102a5aeab379905c6d5d8d7095cd270d679eb3a4019a72" gracePeriod=30 Nov 01 06:12:07.937934 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:07.937871 5040 patch_prober.go:29] interesting pod/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 06:12:07.938407 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:07.937970 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" podUID="e1442af5-c9a0-4995-be20-1f65aa32d0bd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 06:12:08.593749 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:08.593684 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" Nov 01 06:12:08.708763 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:08.708729 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1442af5-c9a0-4995-be20-1f65aa32d0bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1442af5-c9a0-4995-be20-1f65aa32d0bd-istio-envoy\") pod \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\" (UID: \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\") " Nov 01 06:12:08.708975 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:08.708835 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1442af5-c9a0-4995-be20-1f65aa32d0bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1442af5-c9a0-4995-be20-1f65aa32d0bd-credential-socket\") pod \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\" (UID: \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\") " Nov 01 06:12:08.708975 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:08.708869 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1442af5-c9a0-4995-be20-1f65aa32d0bd-istio-token\" (UniqueName: \"kubernetes.io/projected/e1442af5-c9a0-4995-be20-1f65aa32d0bd-istio-token\") pod \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\" (UID: \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\") " Nov 01 06:12:08.708975 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:08.708930 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1442af5-c9a0-4995-be20-1f65aa32d0bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1442af5-c9a0-4995-be20-1f65aa32d0bd-istio-podinfo\") pod \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\" (UID: \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\") " Nov 01 06:12:08.708975 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:08.708962 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1442af5-c9a0-4995-be20-1f65aa32d0bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1442af5-c9a0-4995-be20-1f65aa32d0bd-workload-socket\") pod \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\" (UID: \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\") " Nov 01 06:12:08.709218 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:08.708992 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1442af5-c9a0-4995-be20-1f65aa32d0bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1442af5-c9a0-4995-be20-1f65aa32d0bd-workload-certs\") pod \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\" (UID: \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\") " Nov 01 06:12:08.709218 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:08.709032 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1442af5-c9a0-4995-be20-1f65aa32d0bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1442af5-c9a0-4995-be20-1f65aa32d0bd-istio-data\") pod \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\" (UID: \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\") " Nov 01 06:12:08.709218 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:08.709103 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1442af5-c9a0-4995-be20-1f65aa32d0bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1442af5-c9a0-4995-be20-1f65aa32d0bd-istiod-ca-cert\") pod \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\" (UID: \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\") " Nov 01 06:12:08.709218 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:08.709164 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1442af5-c9a0-4995-be20-1f65aa32d0bd-credential-socket" pod "e1442af5-c9a0-4995-be20-1f65aa32d0bd" (UID: "e1442af5-c9a0-4995-be20-1f65aa32d0bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:12:08.709440 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:08.709220 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1442af5-c9a0-4995-be20-1f65aa32d0bd-kube-api-access-lpdx7\" (UniqueName: \"kubernetes.io/projected/e1442af5-c9a0-4995-be20-1f65aa32d0bd-kube-api-access-lpdx7\") pod \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\" (UID: \"e1442af5-c9a0-4995-be20-1f65aa32d0bd\") " Nov 01 06:12:08.709440 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:08.709240 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1442af5-c9a0-4995-be20-1f65aa32d0bd-workload-socket" pod "e1442af5-c9a0-4995-be20-1f65aa32d0bd" (UID: "e1442af5-c9a0-4995-be20-1f65aa32d0bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:12:08.709440 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:08.709389 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1442af5-c9a0-4995-be20-1f65aa32d0bd-workload-certs" pod "e1442af5-c9a0-4995-be20-1f65aa32d0bd" (UID: "e1442af5-c9a0-4995-be20-1f65aa32d0bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:12:08.709440 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:08.709402 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1442af5-c9a0-4995-be20-1f65aa32d0bd-istio-data" pod "e1442af5-c9a0-4995-be20-1f65aa32d0bd" (UID: "e1442af5-c9a0-4995-be20-1f65aa32d0bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:12:08.709705 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:08.709575 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1442af5-c9a0-4995-be20-1f65aa32d0bd-istiod-ca-cert" pod "e1442af5-c9a0-4995-be20-1f65aa32d0bd" (UID: "e1442af5-c9a0-4995-be20-1f65aa32d0bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:12:08.709765 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:08.709750 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1442af5-c9a0-4995-be20-1f65aa32d0bd-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:12:08.709826 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:08.709765 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1442af5-c9a0-4995-be20-1f65aa32d0bd-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:12:08.709826 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:08.709774 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1442af5-c9a0-4995-be20-1f65aa32d0bd-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:12:08.709826 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:08.709787 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1442af5-c9a0-4995-be20-1f65aa32d0bd-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:12:08.709826 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:08.709799 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1442af5-c9a0-4995-be20-1f65aa32d0bd-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:12:08.711172 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:08.711137 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1442af5-c9a0-4995-be20-1f65aa32d0bd-istio-envoy" pod "e1442af5-c9a0-4995-be20-1f65aa32d0bd" (UID: "e1442af5-c9a0-4995-be20-1f65aa32d0bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:12:08.711724 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:08.711700 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1442af5-c9a0-4995-be20-1f65aa32d0bd-istio-podinfo" pod "e1442af5-c9a0-4995-be20-1f65aa32d0bd" (UID: "e1442af5-c9a0-4995-be20-1f65aa32d0bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:12:08.711815 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:08.711728 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1442af5-c9a0-4995-be20-1f65aa32d0bd-istio-token" pod "e1442af5-c9a0-4995-be20-1f65aa32d0bd" (UID: "e1442af5-c9a0-4995-be20-1f65aa32d0bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:12:08.711871 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:08.711826 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1442af5-c9a0-4995-be20-1f65aa32d0bd-kube-api-access-lpdx7" pod "e1442af5-c9a0-4995-be20-1f65aa32d0bd" (UID: "e1442af5-c9a0-4995-be20-1f65aa32d0bd"). InnerVolumeSpecName "kube-api-access-lpdx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:12:08.807874 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:08.807834 5040 generic.go:356] "Generic (PLEG): container finished" podID="e1442af5-c9a0-4995-be20-1f65aa32d0bd" containerID="782f05aa12faf7e0e9102a5aeab379905c6d5d8d7095cd270d679eb3a4019a72" exitCode=0 Nov 01 06:12:08.808051 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:08.807887 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" Nov 01 06:12:08.808051 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:08.807930 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" event={"ID":"e1442af5-c9a0-4995-be20-1f65aa32d0bd","Type":"ContainerDied","Data":"782f05aa12faf7e0e9102a5aeab379905c6d5d8d7095cd270d679eb3a4019a72"} Nov 01 06:12:08.808051 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:08.808031 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26" event={"ID":"e1442af5-c9a0-4995-be20-1f65aa32d0bd","Type":"ContainerDied","Data":"6e895b55a77bcb284f5f21015996cdf80e082010c583d700bf89931967f38564"} Nov 01 06:12:08.808187 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:08.808065 5040 scope.go:122] "RemoveContainer" containerID="782f05aa12faf7e0e9102a5aeab379905c6d5d8d7095cd270d679eb3a4019a72" Nov 01 06:12:08.810919 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:08.810893 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpdx7\" (UniqueName: \"kubernetes.io/projected/e1442af5-c9a0-4995-be20-1f65aa32d0bd-kube-api-access-lpdx7\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:12:08.811043 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:08.810924 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1442af5-c9a0-4995-be20-1f65aa32d0bd-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:12:08.811043 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:08.810937 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1442af5-c9a0-4995-be20-1f65aa32d0bd-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:12:08.811043 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:08.810952 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1442af5-c9a0-4995-be20-1f65aa32d0bd-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:12:08.822979 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:08.822953 5040 scope.go:122] "RemoveContainer" containerID="782f05aa12faf7e0e9102a5aeab379905c6d5d8d7095cd270d679eb3a4019a72" Nov 01 06:12:08.823440 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:12:08.823368 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"782f05aa12faf7e0e9102a5aeab379905c6d5d8d7095cd270d679eb3a4019a72\": container with ID starting with 782f05aa12faf7e0e9102a5aeab379905c6d5d8d7095cd270d679eb3a4019a72 not found: ID does not exist" containerID="782f05aa12faf7e0e9102a5aeab379905c6d5d8d7095cd270d679eb3a4019a72" Nov 01 06:12:08.823440 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:08.823406 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"782f05aa12faf7e0e9102a5aeab379905c6d5d8d7095cd270d679eb3a4019a72"} err="failed to get container status \"782f05aa12faf7e0e9102a5aeab379905c6d5d8d7095cd270d679eb3a4019a72\": rpc error: code = NotFound desc = could not find container \"782f05aa12faf7e0e9102a5aeab379905c6d5d8d7095cd270d679eb3a4019a72\": container with ID starting with 782f05aa12faf7e0e9102a5aeab379905c6d5d8d7095cd270d679eb3a4019a72 not found: ID does not exist" Nov 01 06:12:08.976322 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:08.976210 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26"] Nov 01 06:12:08.980370 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:08.980344 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-snm26"] Nov 01 06:12:09.516341 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:09.516310 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1442af5-c9a0-4995-be20-1f65aa32d0bd" path="/var/lib/kubelet/pods/e1442af5-c9a0-4995-be20-1f65aa32d0bd/volumes" Nov 01 06:12:09.974409 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:12:09.974323 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:12:12.247164 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:12.247131 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-7d966b6c54-v7flb"] Nov 01 06:12:12.247743 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:12.247387 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-7d966b6c54-v7flb" podUID="07af255e-cc07-4e90-bdba-828ac197fa53" containerName="payload-processing" containerID="cri-o://6aa26c5f51e3c152595122f9649d66fb3763dc5a6554f0a5d62d5f6966206f7e" gracePeriod=30 Nov 01 06:12:12.533905 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:12.533878 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-7d966b6c54-v7flb" Nov 01 06:12:12.654493 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:12.654451 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07af255e-cc07-4e90-bdba-828ac197fa53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07af255e-cc07-4e90-bdba-828ac197fa53-plugins-config-volume\") pod \"07af255e-cc07-4e90-bdba-828ac197fa53\" (UID: \"07af255e-cc07-4e90-bdba-828ac197fa53\") " Nov 01 06:12:12.654695 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:12.654680 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07af255e-cc07-4e90-bdba-828ac197fa53-kube-api-access-p9ltd\" (UniqueName: \"kubernetes.io/projected/07af255e-cc07-4e90-bdba-828ac197fa53-kube-api-access-p9ltd\") pod \"07af255e-cc07-4e90-bdba-828ac197fa53\" (UID: \"07af255e-cc07-4e90-bdba-828ac197fa53\") " Nov 01 06:12:12.655255 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:12.655223 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07af255e-cc07-4e90-bdba-828ac197fa53-plugins-config-volume" pod "07af255e-cc07-4e90-bdba-828ac197fa53" (UID: "07af255e-cc07-4e90-bdba-828ac197fa53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:12:12.657149 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:12.657117 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07af255e-cc07-4e90-bdba-828ac197fa53-kube-api-access-p9ltd" pod "07af255e-cc07-4e90-bdba-828ac197fa53" (UID: "07af255e-cc07-4e90-bdba-828ac197fa53"). InnerVolumeSpecName "kube-api-access-p9ltd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:12:12.756101 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:12.756060 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9ltd\" (UniqueName: \"kubernetes.io/projected/07af255e-cc07-4e90-bdba-828ac197fa53-kube-api-access-p9ltd\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:12:12.756101 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:12.756099 5040 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07af255e-cc07-4e90-bdba-828ac197fa53-plugins-config-volume\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:12:12.846726 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:12.846322 5040 generic.go:356] "Generic (PLEG): container finished" podID="07af255e-cc07-4e90-bdba-828ac197fa53" containerID="6aa26c5f51e3c152595122f9649d66fb3763dc5a6554f0a5d62d5f6966206f7e" exitCode=0 Nov 01 06:12:12.846726 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:12.846361 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-7d966b6c54-v7flb" Nov 01 06:12:12.846726 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:12.846399 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-7d966b6c54-v7flb" event={"ID":"07af255e-cc07-4e90-bdba-828ac197fa53","Type":"ContainerDied","Data":"6aa26c5f51e3c152595122f9649d66fb3763dc5a6554f0a5d62d5f6966206f7e"} Nov 01 06:12:12.846726 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:12.846471 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-7d966b6c54-v7flb" event={"ID":"07af255e-cc07-4e90-bdba-828ac197fa53","Type":"ContainerDied","Data":"dd89aedf4d9bd3f41b1056cef28e1935a6ce366c421cb935f981ac2389e1a194"} Nov 01 06:12:12.846726 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:12.846545 5040 scope.go:122] "RemoveContainer" containerID="6aa26c5f51e3c152595122f9649d66fb3763dc5a6554f0a5d62d5f6966206f7e" Nov 01 06:12:12.861563 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:12.861537 5040 scope.go:122] "RemoveContainer" containerID="6aa26c5f51e3c152595122f9649d66fb3763dc5a6554f0a5d62d5f6966206f7e" Nov 01 06:12:12.861954 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:12:12.861935 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aa26c5f51e3c152595122f9649d66fb3763dc5a6554f0a5d62d5f6966206f7e\": container with ID starting with 6aa26c5f51e3c152595122f9649d66fb3763dc5a6554f0a5d62d5f6966206f7e not found: ID does not exist" containerID="6aa26c5f51e3c152595122f9649d66fb3763dc5a6554f0a5d62d5f6966206f7e" Nov 01 06:12:12.862010 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:12.861967 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aa26c5f51e3c152595122f9649d66fb3763dc5a6554f0a5d62d5f6966206f7e"} err="failed to get container status \"6aa26c5f51e3c152595122f9649d66fb3763dc5a6554f0a5d62d5f6966206f7e\": rpc error: code = NotFound desc = could not find container \"6aa26c5f51e3c152595122f9649d66fb3763dc5a6554f0a5d62d5f6966206f7e\": container with ID starting with 6aa26c5f51e3c152595122f9649d66fb3763dc5a6554f0a5d62d5f6966206f7e not found: ID does not exist" Nov 01 06:12:13.013975 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:13.013940 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-7d966b6c54-v7flb"] Nov 01 06:12:13.017497 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:13.017459 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-7d966b6c54-v7flb"] Nov 01 06:12:13.518057 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:13.518020 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07af255e-cc07-4e90-bdba-828ac197fa53" path="/var/lib/kubelet/pods/07af255e-cc07-4e90-bdba-828ac197fa53/volumes" Nov 01 06:12:16.049594 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:16.049552 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc1c06e9-t8h55"] Nov 01 06:12:16.099949 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:16.099910 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc1c06e9-t8h55"] Nov 01 06:12:17.532740 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:17.532687 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0de30789-420b-4616-a383-b4d9a571e0af" path="/var/lib/kubelet/pods/0de30789-420b-4616-a383-b4d9a571e0af/volumes" Nov 01 06:12:22.139349 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:22.139316 5040 scope.go:122] "RemoveContainer" containerID="709b7d54a7725413ef448845c4dbfd191a34350d2fc69a8f17d6840e013af177" Nov 01 06:12:22.149196 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:22.149175 5040 scope.go:122] "RemoveContainer" containerID="1c380dfa9c6d6ee8dae542b8325be07e3d8fcb13ed154004b5cb4d45adb63125" Nov 01 06:12:22.159170 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:22.159148 5040 scope.go:122] "RemoveContainer" containerID="83bf0052bd200b3c42c86db55737b383e70c43e85dc4d47744304afd58fb9a3c" Nov 01 06:12:22.168687 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:22.168664 5040 scope.go:122] "RemoveContainer" containerID="acdf6aa47dd3b4eff1b841a8dacef5d3910ba952af6f251b786fff8bd62d4c92" Nov 01 06:12:22.177834 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:22.177813 5040 scope.go:122] "RemoveContainer" containerID="28e8d7041c573bab3e7da6734f6dd1e52720cf936b546534a7bcfd7198bf481b" Nov 01 06:12:22.187458 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:22.187437 5040 scope.go:122] "RemoveContainer" containerID="32fdafba2fa6ab689859aa6ef413ba3679ddce74b72ca568228b9a34e5586d20" Nov 01 06:12:29.049901 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:29.049861 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7da6a9db-zgk6m"] Nov 01 06:12:29.100196 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:29.100162 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7da6a9db-zgk6m"] Nov 01 06:12:29.515459 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:29.515423 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05a3c38e-d470-4e9b-b444-20730fa25f16" path="/var/lib/kubelet/pods/05a3c38e-d470-4e9b-b444-20730fa25f16/volumes" Nov 01 06:12:32.714605 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.714562 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6"] Nov 01 06:12:32.715316 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.715292 5040 cpu_manager.go:414] "containerMap: removing container" podUID="a8b8a332-acb6-4075-a630-cc2f89ea7048" containerName="revoke-keys" Nov 01 06:12:32.715406 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.715318 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8b8a332-acb6-4075-a630-cc2f89ea7048" containerName="revoke-keys" Nov 01 06:12:32.715406 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.715332 5040 cpu_manager.go:414] "containerMap: removing container" podUID="a325a624-c22a-4425-9002-ad68a3c22314" containerName="istio-proxy" Nov 01 06:12:32.715406 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.715339 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a325a624-c22a-4425-9002-ad68a3c22314" containerName="istio-proxy" Nov 01 06:12:32.715406 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.715363 5040 cpu_manager.go:414] "containerMap: removing container" podUID="e1442af5-c9a0-4995-be20-1f65aa32d0bd" containerName="istio-proxy" Nov 01 06:12:32.715406 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.715371 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1442af5-c9a0-4995-be20-1f65aa32d0bd" containerName="istio-proxy" Nov 01 06:12:32.715406 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.715389 5040 cpu_manager.go:414] "containerMap: removing container" podUID="0073efde-c913-424b-93e8-bda25a24a209" containerName="istio-proxy" Nov 01 06:12:32.715406 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.715397 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0073efde-c913-424b-93e8-bda25a24a209" containerName="istio-proxy" Nov 01 06:12:32.715787 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.715414 5040 cpu_manager.go:414] "containerMap: removing container" podUID="7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd" containerName="payload-pre-processing" Nov 01 06:12:32.715787 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.715422 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd" containerName="payload-pre-processing" Nov 01 06:12:32.715787 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.715434 5040 cpu_manager.go:414] "containerMap: removing container" podUID="a325a624-c22a-4425-9002-ad68a3c22314" containerName="istio-proxy" Nov 01 06:12:32.715787 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.715439 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a325a624-c22a-4425-9002-ad68a3c22314" containerName="istio-proxy" Nov 01 06:12:32.715787 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.715461 5040 cpu_manager.go:414] "containerMap: removing container" podUID="0073efde-c913-424b-93e8-bda25a24a209" containerName="istio-proxy" Nov 01 06:12:32.715787 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.715466 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0073efde-c913-424b-93e8-bda25a24a209" containerName="istio-proxy" Nov 01 06:12:32.715787 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.715472 5040 cpu_manager.go:414] "containerMap: removing container" podUID="feb6d45c-3a57-4a1e-863c-8c0d7972d453" containerName="payload-processing" Nov 01 06:12:32.715787 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.715493 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feb6d45c-3a57-4a1e-863c-8c0d7972d453" containerName="payload-processing" Nov 01 06:12:32.715787 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.715504 5040 cpu_manager.go:414] "containerMap: removing container" podUID="07af255e-cc07-4e90-bdba-828ac197fa53" containerName="payload-processing" Nov 01 06:12:32.715787 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.715508 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07af255e-cc07-4e90-bdba-828ac197fa53" containerName="payload-processing" Nov 01 06:12:32.715787 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.715603 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1442af5-c9a0-4995-be20-1f65aa32d0bd" containerName="istio-proxy" Nov 01 06:12:32.715787 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.715612 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="0073efde-c913-424b-93e8-bda25a24a209" containerName="istio-proxy" Nov 01 06:12:32.715787 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.715618 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b7efe8a-80d1-4b2c-af1b-ae9cbf99d4fd" containerName="payload-pre-processing" Nov 01 06:12:32.715787 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.715627 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="a325a624-c22a-4425-9002-ad68a3c22314" containerName="istio-proxy" Nov 01 06:12:32.715787 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.715634 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="0073efde-c913-424b-93e8-bda25a24a209" containerName="istio-proxy" Nov 01 06:12:32.715787 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.715640 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="feb6d45c-3a57-4a1e-863c-8c0d7972d453" containerName="payload-processing" Nov 01 06:12:32.715787 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.715646 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8b8a332-acb6-4075-a630-cc2f89ea7048" containerName="revoke-keys" Nov 01 06:12:32.715787 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.715652 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="07af255e-cc07-4e90-bdba-828ac197fa53" containerName="payload-processing" Nov 01 06:12:32.715787 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.715658 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="a325a624-c22a-4425-9002-ad68a3c22314" containerName="istio-proxy" Nov 01 06:12:32.721289 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.721266 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6" Nov 01 06:12:32.770257 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.770224 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6"] Nov 01 06:12:32.816654 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.816619 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-3578ce\"/\"default-dockercfg-s6gdj\"" Nov 01 06:12:32.816654 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.816628 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-3578ce\"/\"kube-root-ca.crt\"" Nov 01 06:12:32.816856 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.816624 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2fbb701-9557-4fc1-a149-591122b79492-tls-certs\") pod \"praxis-route-3578ce-kserve-57f8955b99-jhqf6\" (UID: \"c2fbb701-9557-4fc1-a149-591122b79492\") " pod="ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6" Nov 01 06:12:32.816856 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.816620 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-3578ce\"/\"openshift-service-ca.crt\"" Nov 01 06:12:32.816856 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.816717 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvh59\" (UniqueName: \"kubernetes.io/projected/c2fbb701-9557-4fc1-a149-591122b79492-kube-api-access-pvh59\") pod \"praxis-route-3578ce-kserve-57f8955b99-jhqf6\" (UID: \"c2fbb701-9557-4fc1-a149-591122b79492\") " pod="ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6" Nov 01 06:12:32.816856 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.816831 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2fbb701-9557-4fc1-a149-591122b79492-dshm\") pod \"praxis-route-3578ce-kserve-57f8955b99-jhqf6\" (UID: \"c2fbb701-9557-4fc1-a149-591122b79492\") " pod="ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6" Nov 01 06:12:32.816981 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.816898 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2fbb701-9557-4fc1-a149-591122b79492-home\") pod \"praxis-route-3578ce-kserve-57f8955b99-jhqf6\" (UID: \"c2fbb701-9557-4fc1-a149-591122b79492\") " pod="ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6" Nov 01 06:12:32.816981 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.816936 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2fbb701-9557-4fc1-a149-591122b79492-model-cache\") pod \"praxis-route-3578ce-kserve-57f8955b99-jhqf6\" (UID: \"c2fbb701-9557-4fc1-a149-591122b79492\") " pod="ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6" Nov 01 06:12:32.817084 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.816991 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2fbb701-9557-4fc1-a149-591122b79492-tmp-dir\") pod \"praxis-route-3578ce-kserve-57f8955b99-jhqf6\" (UID: \"c2fbb701-9557-4fc1-a149-591122b79492\") " pod="ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6" Nov 01 06:12:32.824728 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.824709 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-3578ce\"/\"praxis-route-3578ce-kserve-self-signed-certs\"" Nov 01 06:12:32.918647 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.918597 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2fbb701-9557-4fc1-a149-591122b79492-model-cache\") pod \"praxis-route-3578ce-kserve-57f8955b99-jhqf6\" (UID: \"c2fbb701-9557-4fc1-a149-591122b79492\") " pod="ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6" Nov 01 06:12:32.918647 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.918640 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2fbb701-9557-4fc1-a149-591122b79492-tmp-dir\") pod \"praxis-route-3578ce-kserve-57f8955b99-jhqf6\" (UID: \"c2fbb701-9557-4fc1-a149-591122b79492\") " pod="ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6" Nov 01 06:12:32.918897 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.918783 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2fbb701-9557-4fc1-a149-591122b79492-tls-certs\") pod \"praxis-route-3578ce-kserve-57f8955b99-jhqf6\" (UID: \"c2fbb701-9557-4fc1-a149-591122b79492\") " pod="ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6" Nov 01 06:12:32.918897 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.918818 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvh59\" (UniqueName: \"kubernetes.io/projected/c2fbb701-9557-4fc1-a149-591122b79492-kube-api-access-pvh59\") pod \"praxis-route-3578ce-kserve-57f8955b99-jhqf6\" (UID: \"c2fbb701-9557-4fc1-a149-591122b79492\") " pod="ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6" Nov 01 06:12:32.919008 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.918907 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2fbb701-9557-4fc1-a149-591122b79492-dshm\") pod \"praxis-route-3578ce-kserve-57f8955b99-jhqf6\" (UID: \"c2fbb701-9557-4fc1-a149-591122b79492\") " pod="ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6" Nov 01 06:12:32.919008 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.918950 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2fbb701-9557-4fc1-a149-591122b79492-home\") pod \"praxis-route-3578ce-kserve-57f8955b99-jhqf6\" (UID: \"c2fbb701-9557-4fc1-a149-591122b79492\") " pod="ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6" Nov 01 06:12:32.919165 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.919073 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2fbb701-9557-4fc1-a149-591122b79492-model-cache\") pod \"praxis-route-3578ce-kserve-57f8955b99-jhqf6\" (UID: \"c2fbb701-9557-4fc1-a149-591122b79492\") " pod="ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6" Nov 01 06:12:32.919245 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.919181 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2fbb701-9557-4fc1-a149-591122b79492-tmp-dir\") pod \"praxis-route-3578ce-kserve-57f8955b99-jhqf6\" (UID: \"c2fbb701-9557-4fc1-a149-591122b79492\") " pod="ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6" Nov 01 06:12:32.919313 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.919290 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2fbb701-9557-4fc1-a149-591122b79492-home\") pod \"praxis-route-3578ce-kserve-57f8955b99-jhqf6\" (UID: \"c2fbb701-9557-4fc1-a149-591122b79492\") " pod="ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6" Nov 01 06:12:32.921466 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.921433 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2fbb701-9557-4fc1-a149-591122b79492-dshm\") pod \"praxis-route-3578ce-kserve-57f8955b99-jhqf6\" (UID: \"c2fbb701-9557-4fc1-a149-591122b79492\") " pod="ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6" Nov 01 06:12:32.922239 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.922213 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2fbb701-9557-4fc1-a149-591122b79492-tls-certs\") pod \"praxis-route-3578ce-kserve-57f8955b99-jhqf6\" (UID: \"c2fbb701-9557-4fc1-a149-591122b79492\") " pod="ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6" Nov 01 06:12:32.980783 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:32.980714 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvh59\" (UniqueName: \"kubernetes.io/projected/c2fbb701-9557-4fc1-a149-591122b79492-kube-api-access-pvh59\") pod \"praxis-route-3578ce-kserve-57f8955b99-jhqf6\" (UID: \"c2fbb701-9557-4fc1-a149-591122b79492\") " pod="ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6" Nov 01 06:12:33.035106 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:33.035071 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6" Nov 01 06:12:33.418429 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:12:33.418385 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2fbb701_9557_4fc1_a149_591122b79492.slice/crio-ec45abbae19ff172000bba60982cd0f2003c756efc316f4e0983492aa8ad934e WatchSource:0}: Error finding container ec45abbae19ff172000bba60982cd0f2003c756efc316f4e0983492aa8ad934e: Status 404 returned error can't find the container with id ec45abbae19ff172000bba60982cd0f2003c756efc316f4e0983492aa8ad934e Nov 01 06:12:33.418584 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:33.418556 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6"] Nov 01 06:12:34.000071 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:34.000028 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6" event={"ID":"c2fbb701-9557-4fc1-a149-591122b79492","Type":"ContainerStarted","Data":"8da601a66fe3d5783d1e8f618a7a0c4bf0127ba87cfbdee37c7e6ea0ebe45c69"} Nov 01 06:12:34.000590 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:34.000085 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6" event={"ID":"c2fbb701-9557-4fc1-a149-591122b79492","Type":"ContainerStarted","Data":"ec45abbae19ff172000bba60982cd0f2003c756efc316f4e0983492aa8ad934e"} Nov 01 06:12:34.035565 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:34.035531 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6" Nov 01 06:12:34.119174 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:34.119117 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6" podStartSLOduration=2.119104997 podStartE2EDuration="2.119104997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:12:34.117240817 +0000 UTC m=+1632.772357985" watchObservedRunningTime="2026-11-01 06:12:34.119104997 +0000 UTC m=+1632.774222163" Nov 01 06:12:35.049875 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:35.049831 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cde7b767-jkxfz"] Nov 01 06:12:35.099292 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:35.099256 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cde7b767-jkxfz"] Nov 01 06:12:35.515583 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:35.515553 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64837f7d-6b63-40de-a034-35701ceda44f" path="/var/lib/kubelet/pods/64837f7d-6b63-40de-a034-35701ceda44f/volumes" Nov 01 06:12:39.974906 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:12:39.974856 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:12:43.036029 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:43.035983 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6" Nov 01 06:12:43.048993 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:43.048963 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6" Nov 01 06:12:43.072219 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:43.072188 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6" Nov 01 06:12:53.050832 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:53.050798 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e15fe852-pcqk4"] Nov 01 06:12:53.100432 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:53.100386 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e15fe852-pcqk4"] Nov 01 06:12:53.514974 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:12:53.514942 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="200b968b-65f6-4f89-9132-9caf0ff4e291" path="/var/lib/kubelet/pods/200b968b-65f6-4f89-9132-9caf0ff4e291/volumes" Nov 01 06:13:01.047834 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:01.047754 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df8051bb-2cnn2"] Nov 01 06:13:01.097695 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:01.097659 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df8051bb-2cnn2"] Nov 01 06:13:01.515670 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:01.515639 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae460a49-f48c-48a9-b0be-a4d5d096411f" path="/var/lib/kubelet/pods/ae460a49-f48c-48a9-b0be-a4d5d096411f/volumes" Nov 01 06:13:02.687226 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:02.687190 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq"] Nov 01 06:13:02.692238 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:02.692217 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" Nov 01 06:13:02.746326 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:02.746293 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-9c8fc5\"/\"kube-root-ca.crt\"" Nov 01 06:13:02.766882 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:02.766836 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5daa03ca-824b-4799-a11b-4b3af1512c38-model-cache\") pod \"distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq\" (UID: \"5daa03ca-824b-4799-a11b-4b3af1512c38\") " pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" Nov 01 06:13:02.767061 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:02.766991 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvzj9\" (UniqueName: \"kubernetes.io/projected/5daa03ca-824b-4799-a11b-4b3af1512c38-kube-api-access-dvzj9\") pod \"distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq\" (UID: \"5daa03ca-824b-4799-a11b-4b3af1512c38\") " pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" Nov 01 06:13:02.767061 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:02.767026 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5daa03ca-824b-4799-a11b-4b3af1512c38-dshm\") pod \"distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq\" (UID: \"5daa03ca-824b-4799-a11b-4b3af1512c38\") " pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" Nov 01 06:13:02.767196 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:02.767129 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5daa03ca-824b-4799-a11b-4b3af1512c38-tmp-dir\") pod \"distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq\" (UID: \"5daa03ca-824b-4799-a11b-4b3af1512c38\") " pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" Nov 01 06:13:02.767196 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:02.767165 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5daa03ca-824b-4799-a11b-4b3af1512c38-tls-certs\") pod \"distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq\" (UID: \"5daa03ca-824b-4799-a11b-4b3af1512c38\") " pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" Nov 01 06:13:02.767309 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:02.767277 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5daa03ca-824b-4799-a11b-4b3af1512c38-home\") pod \"distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq\" (UID: \"5daa03ca-824b-4799-a11b-4b3af1512c38\") " pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" Nov 01 06:13:02.796949 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:02.796915 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9c8fc5\"/\"distinct-w0-9c8fc5-kserve-self-signed-certs\"" Nov 01 06:13:02.796949 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:02.796915 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-9c8fc5\"/\"openshift-service-ca.crt\"" Nov 01 06:13:02.796949 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:02.796942 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9c8fc5\"/\"default-dockercfg-rjmhh\"" Nov 01 06:13:02.800326 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:02.800308 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq"] Nov 01 06:13:02.868126 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:02.868082 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvzj9\" (UniqueName: \"kubernetes.io/projected/5daa03ca-824b-4799-a11b-4b3af1512c38-kube-api-access-dvzj9\") pod \"distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq\" (UID: \"5daa03ca-824b-4799-a11b-4b3af1512c38\") " pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" Nov 01 06:13:02.868332 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:02.868175 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5daa03ca-824b-4799-a11b-4b3af1512c38-dshm\") pod \"distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq\" (UID: \"5daa03ca-824b-4799-a11b-4b3af1512c38\") " pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" Nov 01 06:13:02.869357 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:02.869296 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5daa03ca-824b-4799-a11b-4b3af1512c38-tmp-dir\") pod \"distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq\" (UID: \"5daa03ca-824b-4799-a11b-4b3af1512c38\") " pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" Nov 01 06:13:02.869547 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:02.869378 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5daa03ca-824b-4799-a11b-4b3af1512c38-tls-certs\") pod \"distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq\" (UID: \"5daa03ca-824b-4799-a11b-4b3af1512c38\") " pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" Nov 01 06:13:02.869547 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:02.869448 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5daa03ca-824b-4799-a11b-4b3af1512c38-home\") pod \"distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq\" (UID: \"5daa03ca-824b-4799-a11b-4b3af1512c38\") " pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" Nov 01 06:13:02.869677 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:02.869555 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5daa03ca-824b-4799-a11b-4b3af1512c38-model-cache\") pod \"distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq\" (UID: \"5daa03ca-824b-4799-a11b-4b3af1512c38\") " pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" Nov 01 06:13:02.869966 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:02.869942 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5daa03ca-824b-4799-a11b-4b3af1512c38-model-cache\") pod \"distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq\" (UID: \"5daa03ca-824b-4799-a11b-4b3af1512c38\") " pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" Nov 01 06:13:02.871142 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:02.870874 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5daa03ca-824b-4799-a11b-4b3af1512c38-home\") pod \"distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq\" (UID: \"5daa03ca-824b-4799-a11b-4b3af1512c38\") " pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" Nov 01 06:13:02.871142 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:02.870925 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5daa03ca-824b-4799-a11b-4b3af1512c38-tmp-dir\") pod \"distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq\" (UID: \"5daa03ca-824b-4799-a11b-4b3af1512c38\") " pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" Nov 01 06:13:02.871729 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:02.871707 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5daa03ca-824b-4799-a11b-4b3af1512c38-dshm\") pod \"distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq\" (UID: \"5daa03ca-824b-4799-a11b-4b3af1512c38\") " pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" Nov 01 06:13:02.873258 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:02.873236 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5daa03ca-824b-4799-a11b-4b3af1512c38-tls-certs\") pod \"distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq\" (UID: \"5daa03ca-824b-4799-a11b-4b3af1512c38\") " pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" Nov 01 06:13:02.930435 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:02.930400 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvzj9\" (UniqueName: \"kubernetes.io/projected/5daa03ca-824b-4799-a11b-4b3af1512c38-kube-api-access-dvzj9\") pod \"distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq\" (UID: \"5daa03ca-824b-4799-a11b-4b3af1512c38\") " pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" Nov 01 06:13:03.005600 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:03.005567 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" Nov 01 06:13:03.453942 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:03.453908 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq"] Nov 01 06:13:03.454370 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:13:03.454344 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5daa03ca_824b_4799_a11b_4b3af1512c38.slice/crio-28f135eef831d1b96eb4b72a9e5fb689928c8c8b87a2a839c9026387d6178aae WatchSource:0}: Error finding container 28f135eef831d1b96eb4b72a9e5fb689928c8c8b87a2a839c9026387d6178aae: Status 404 returned error can't find the container with id 28f135eef831d1b96eb4b72a9e5fb689928c8c8b87a2a839c9026387d6178aae Nov 01 06:13:04.048538 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:04.048502 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d981b767-hx96n"] Nov 01 06:13:04.098677 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:04.098645 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d981b767-hx96n"] Nov 01 06:13:04.211202 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:04.211165 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" event={"ID":"5daa03ca-824b-4799-a11b-4b3af1512c38","Type":"ContainerStarted","Data":"2b56028a64eec80ff09aa3ce9ffca497844c5ff3cf39634cbe2e6c6e7026823e"} Nov 01 06:13:04.211364 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:04.211207 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" event={"ID":"5daa03ca-824b-4799-a11b-4b3af1512c38","Type":"ContainerStarted","Data":"28f135eef831d1b96eb4b72a9e5fb689928c8c8b87a2a839c9026387d6178aae"} Nov 01 06:13:04.329451 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:04.329338 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" podStartSLOduration=2.329325198 podStartE2EDuration="2.329325198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:13:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:13:04.328602051 +0000 UTC m=+1662.983719219" watchObservedRunningTime="2026-11-01 06:13:04.329325198 +0000 UTC m=+1662.984442365" Nov 01 06:13:05.006297 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:05.006266 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" Nov 01 06:13:05.515245 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:05.515211 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2ee8964-c194-4d28-94d1-7518f2e69e15" path="/var/lib/kubelet/pods/a2ee8964-c194-4d28-94d1-7518f2e69e15/volumes" Nov 01 06:13:09.973407 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:13:09.973371 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:13:13.006582 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:13.006542 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" Nov 01 06:13:13.019020 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:13.018990 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" Nov 01 06:13:13.285194 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:13.285161 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" Nov 01 06:13:16.048047 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:16.048008 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3721c766-plfmj"] Nov 01 06:13:16.098237 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:16.098200 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3721c766-plfmj"] Nov 01 06:13:17.521405 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:17.521366 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69e53ef2-a284-4996-a030-195fc5d509fa" path="/var/lib/kubelet/pods/69e53ef2-a284-4996-a030-195fc5d509fa/volumes" Nov 01 06:13:22.254771 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:22.254735 5040 scope.go:122] "RemoveContainer" containerID="391befc3245780e53fa36a9f8cdf97744e4943c265c6a350fccc21fdbdc19183" Nov 01 06:13:22.264102 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:22.264079 5040 scope.go:122] "RemoveContainer" containerID="059ceceeda90b0b4496b1a298885e682cfd9a64c1e8fce0611e11eba1c7fff31" Nov 01 06:13:22.273990 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:22.273968 5040 scope.go:122] "RemoveContainer" containerID="51e200a0419a44ff0527b12a1a4f82d566c61dcdafa52e468d3a1039328f59cf" Nov 01 06:13:22.282720 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:22.282702 5040 scope.go:122] "RemoveContainer" containerID="608b817a645129baa1284fef912293636b095803b1444e89a1e7c468842dc8b0" Nov 01 06:13:22.292266 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:22.292230 5040 scope.go:122] "RemoveContainer" containerID="6a8c2d06c7ff9e57a2a8290d656f6c152ed57372ced5fa4529b4d10c62779d6a" Nov 01 06:13:22.301193 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:22.301172 5040 scope.go:122] "RemoveContainer" containerID="f04cc6736afe59ba1350964fe7e8043ab3a2937a6d8f1b368b5d6bddcdb08d39" Nov 01 06:13:29.732212 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:29.732168 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc"] Nov 01 06:13:29.737976 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:29.737957 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc" Nov 01 06:13:29.781012 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:29.780973 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66fdf9da-4790-40c5-9434-052c7465d23e-tls-certs\") pod \"distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc\" (UID: \"66fdf9da-4790-40c5-9434-052c7465d23e\") " pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc" Nov 01 06:13:29.781194 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:29.781065 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66fdf9da-4790-40c5-9434-052c7465d23e-home\") pod \"distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc\" (UID: \"66fdf9da-4790-40c5-9434-052c7465d23e\") " pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc" Nov 01 06:13:29.781194 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:29.781097 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbcft\" (UniqueName: \"kubernetes.io/projected/66fdf9da-4790-40c5-9434-052c7465d23e-kube-api-access-dbcft\") pod \"distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc\" (UID: \"66fdf9da-4790-40c5-9434-052c7465d23e\") " pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc" Nov 01 06:13:29.781284 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:29.781199 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66fdf9da-4790-40c5-9434-052c7465d23e-model-cache\") pod \"distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc\" (UID: \"66fdf9da-4790-40c5-9434-052c7465d23e\") " pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc" Nov 01 06:13:29.781284 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:29.781258 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66fdf9da-4790-40c5-9434-052c7465d23e-tmp-dir\") pod \"distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc\" (UID: \"66fdf9da-4790-40c5-9434-052c7465d23e\") " pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc" Nov 01 06:13:29.781364 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:29.781295 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66fdf9da-4790-40c5-9434-052c7465d23e-dshm\") pod \"distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc\" (UID: \"66fdf9da-4790-40c5-9434-052c7465d23e\") " pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc" Nov 01 06:13:29.794598 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:29.794565 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc"] Nov 01 06:13:29.840884 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:29.840848 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9c8fc5\"/\"distinct-2-w0-9c8fc5-kserve-self-signed-certs\"" Nov 01 06:13:29.882722 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:29.882681 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66fdf9da-4790-40c5-9434-052c7465d23e-tls-certs\") pod \"distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc\" (UID: \"66fdf9da-4790-40c5-9434-052c7465d23e\") " pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc" Nov 01 06:13:29.882931 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:29.882745 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66fdf9da-4790-40c5-9434-052c7465d23e-home\") pod \"distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc\" (UID: \"66fdf9da-4790-40c5-9434-052c7465d23e\") " pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc" Nov 01 06:13:29.882931 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:29.882778 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbcft\" (UniqueName: \"kubernetes.io/projected/66fdf9da-4790-40c5-9434-052c7465d23e-kube-api-access-dbcft\") pod \"distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc\" (UID: \"66fdf9da-4790-40c5-9434-052c7465d23e\") " pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc" Nov 01 06:13:29.882931 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:29.882817 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66fdf9da-4790-40c5-9434-052c7465d23e-model-cache\") pod \"distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc\" (UID: \"66fdf9da-4790-40c5-9434-052c7465d23e\") " pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc" Nov 01 06:13:29.882931 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:29.882889 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66fdf9da-4790-40c5-9434-052c7465d23e-tmp-dir\") pod \"distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc\" (UID: \"66fdf9da-4790-40c5-9434-052c7465d23e\") " pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc" Nov 01 06:13:29.883160 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:29.882941 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66fdf9da-4790-40c5-9434-052c7465d23e-dshm\") pod \"distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc\" (UID: \"66fdf9da-4790-40c5-9434-052c7465d23e\") " pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc" Nov 01 06:13:29.883606 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:29.883577 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66fdf9da-4790-40c5-9434-052c7465d23e-home\") pod \"distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc\" (UID: \"66fdf9da-4790-40c5-9434-052c7465d23e\") " pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc" Nov 01 06:13:29.883737 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:29.883577 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66fdf9da-4790-40c5-9434-052c7465d23e-tmp-dir\") pod \"distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc\" (UID: \"66fdf9da-4790-40c5-9434-052c7465d23e\") " pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc" Nov 01 06:13:29.883938 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:29.883911 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66fdf9da-4790-40c5-9434-052c7465d23e-model-cache\") pod \"distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc\" (UID: \"66fdf9da-4790-40c5-9434-052c7465d23e\") " pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc" Nov 01 06:13:29.886007 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:29.885974 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66fdf9da-4790-40c5-9434-052c7465d23e-dshm\") pod \"distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc\" (UID: \"66fdf9da-4790-40c5-9434-052c7465d23e\") " pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc" Nov 01 06:13:29.886007 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:29.885993 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66fdf9da-4790-40c5-9434-052c7465d23e-tls-certs\") pod \"distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc\" (UID: \"66fdf9da-4790-40c5-9434-052c7465d23e\") " pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc" Nov 01 06:13:29.945008 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:29.944969 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbcft\" (UniqueName: \"kubernetes.io/projected/66fdf9da-4790-40c5-9434-052c7465d23e-kube-api-access-dbcft\") pod \"distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc\" (UID: \"66fdf9da-4790-40c5-9434-052c7465d23e\") " pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc" Nov 01 06:13:30.050190 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:30.050146 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc" Nov 01 06:13:30.465491 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:30.465455 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc"] Nov 01 06:13:30.466523 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:13:30.466497 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66fdf9da_4790_40c5_9434_052c7465d23e.slice/crio-912450bc3f6692e3e119f7795d739290f2007e2c3aeebf6e5fb910e5c025c3d9 WatchSource:0}: Error finding container 912450bc3f6692e3e119f7795d739290f2007e2c3aeebf6e5fb910e5c025c3d9: Status 404 returned error can't find the container with id 912450bc3f6692e3e119f7795d739290f2007e2c3aeebf6e5fb910e5c025c3d9 Nov 01 06:13:31.052827 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:31.052795 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6608305-fjc8z"] Nov 01 06:13:31.102532 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:31.102494 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6608305-fjc8z"] Nov 01 06:13:31.403541 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:31.403450 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc" event={"ID":"66fdf9da-4790-40c5-9434-052c7465d23e","Type":"ContainerStarted","Data":"af58ca6fbb09563c9b43340b1a852a6aa9259ce988e71d84054eee5b812dcfad"} Nov 01 06:13:31.403541 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:31.403512 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc" event={"ID":"66fdf9da-4790-40c5-9434-052c7465d23e","Type":"ContainerStarted","Data":"912450bc3f6692e3e119f7795d739290f2007e2c3aeebf6e5fb910e5c025c3d9"} Nov 01 06:13:31.515036 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:31.515005 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e58e9d0-1122-480f-b5ed-60cadb389f23" path="/var/lib/kubelet/pods/3e58e9d0-1122-480f-b5ed-60cadb389f23/volumes" Nov 01 06:13:31.526428 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:31.526361 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc" podStartSLOduration=2.526343049 podStartE2EDuration="2.526343049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:13:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:13:31.521452727 +0000 UTC m=+1690.176569898" watchObservedRunningTime="2026-11-01 06:13:31.526343049 +0000 UTC m=+1690.181460217" Nov 01 06:13:32.050419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:32.050380 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc" Nov 01 06:13:39.049839 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:39.049803 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp"] Nov 01 06:13:39.050422 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:39.050095 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" podUID="8bf77172-40a2-4853-9e6a-02a0ed2cbc4b" containerName="istio-proxy" containerID="cri-o://e0467cec0d126bfccecd68df4cfbdcc9825d3d2627a83cdeba6c1a4955a9f0fb" gracePeriod=30 Nov 01 06:13:39.100649 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:39.100611 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6"] Nov 01 06:13:39.100927 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:39.100878 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" podUID="eba373b4-ae6e-45af-9833-766ae34813d5" containerName="istio-proxy" containerID="cri-o://6a77c1d0d9ff606c23eee591e1ea26ecf32e7f94bb9037b135d5d8072683b86b" gracePeriod=30 Nov 01 06:13:39.103522 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:39.103497 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb"] Nov 01 06:13:39.103718 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:39.103678 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" podUID="ac11b327-8061-4422-af8d-a37b205d74cf" containerName="istio-proxy" containerID="cri-o://eb313244f18a2917805528aa97996609ad8fe94ceaaba137a498f8eb79798246" gracePeriod=30 Nov 01 06:13:39.152100 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:39.152043 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs"] Nov 01 06:13:39.152309 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:39.152281 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" podUID="8d7c546f-f34b-4824-b012-962a5e7f68c6" containerName="istio-proxy" containerID="cri-o://25047227c2789d294c9a09cb93001f4ad4745caf8d974ed0730b686e2f4d5a41" gracePeriod=30 Nov 01 06:13:39.157070 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:39.156146 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z"] Nov 01 06:13:39.157070 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:39.156364 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" podUID="d6aab7db-e2e7-40ef-bf44-6873237c494f" containerName="istio-proxy" containerID="cri-o://864c5b50992c92f17b5c30b35e8c357b3e03c43378d7fc20601c29802627b0dd" gracePeriod=30 Nov 01 06:13:39.973447 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:13:39.973412 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:13:40.050962 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:40.050925 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc" Nov 01 06:13:40.063094 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:40.063072 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc" Nov 01 06:13:40.486183 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:40.486152 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc" Nov 01 06:13:44.034589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.034548 5040 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 06:13:44.035076 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.034627 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" podUID="8bf77172-40a2-4853-9e6a-02a0ed2cbc4b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 06:13:44.404679 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.404646 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" Nov 01 06:13:44.453520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.451199 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-istio-token\" (UniqueName: \"kubernetes.io/projected/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-istio-token\") pod \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\" (UID: \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\") " Nov 01 06:13:44.453520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.451254 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-workload-certs\") pod \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\" (UID: \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\") " Nov 01 06:13:44.453520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.451633 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-workload-certs" pod "8bf77172-40a2-4853-9e6a-02a0ed2cbc4b" (UID: "8bf77172-40a2-4853-9e6a-02a0ed2cbc4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:13:44.453520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.451637 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-credential-socket\") pod \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\" (UID: \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\") " Nov 01 06:13:44.453520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.451795 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-istio-data\") pod \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\" (UID: \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\") " Nov 01 06:13:44.453520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.451825 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-kube-api-access-kxvx6\" (UniqueName: \"kubernetes.io/projected/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-kube-api-access-kxvx6\") pod \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\" (UID: \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\") " Nov 01 06:13:44.453520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.451825 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-credential-socket" pod "8bf77172-40a2-4853-9e6a-02a0ed2cbc4b" (UID: "8bf77172-40a2-4853-9e6a-02a0ed2cbc4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:13:44.453520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.451847 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-istiod-ca-cert\") pod \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\" (UID: \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\") " Nov 01 06:13:44.453520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.451994 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-workload-socket\") pod \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\" (UID: \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\") " Nov 01 06:13:44.453520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.452027 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-istio-data" pod "8bf77172-40a2-4853-9e6a-02a0ed2cbc4b" (UID: "8bf77172-40a2-4853-9e6a-02a0ed2cbc4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:13:44.453520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.452045 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-istio-envoy\") pod \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\" (UID: \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\") " Nov 01 06:13:44.453520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.452077 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-istio-podinfo\") pod \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\" (UID: \"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b\") " Nov 01 06:13:44.453520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.452080 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-istiod-ca-cert" pod "8bf77172-40a2-4853-9e6a-02a0ed2cbc4b" (UID: "8bf77172-40a2-4853-9e6a-02a0ed2cbc4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:13:44.453520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.452166 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-workload-socket" pod "8bf77172-40a2-4853-9e6a-02a0ed2cbc4b" (UID: "8bf77172-40a2-4853-9e6a-02a0ed2cbc4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:13:44.453520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.452584 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.453520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.452604 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.453520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.452618 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.453520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.452632 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.453520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.452645 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.455711 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.455677 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-istio-podinfo" pod "8bf77172-40a2-4853-9e6a-02a0ed2cbc4b" (UID: "8bf77172-40a2-4853-9e6a-02a0ed2cbc4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:13:44.455820 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.455714 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-istio-token" pod "8bf77172-40a2-4853-9e6a-02a0ed2cbc4b" (UID: "8bf77172-40a2-4853-9e6a-02a0ed2cbc4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:13:44.455820 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.455752 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-istio-envoy" pod "8bf77172-40a2-4853-9e6a-02a0ed2cbc4b" (UID: "8bf77172-40a2-4853-9e6a-02a0ed2cbc4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:13:44.455820 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.455772 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-kube-api-access-kxvx6" pod "8bf77172-40a2-4853-9e6a-02a0ed2cbc4b" (UID: "8bf77172-40a2-4853-9e6a-02a0ed2cbc4b"). InnerVolumeSpecName "kube-api-access-kxvx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:13:44.534031 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.525657 5040 generic.go:356] "Generic (PLEG): container finished" podID="8bf77172-40a2-4853-9e6a-02a0ed2cbc4b" containerID="e0467cec0d126bfccecd68df4cfbdcc9825d3d2627a83cdeba6c1a4955a9f0fb" exitCode=0 Nov 01 06:13:44.534031 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.525753 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" event={"ID":"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b","Type":"ContainerDied","Data":"e0467cec0d126bfccecd68df4cfbdcc9825d3d2627a83cdeba6c1a4955a9f0fb"} Nov 01 06:13:44.534031 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.525801 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" event={"ID":"8bf77172-40a2-4853-9e6a-02a0ed2cbc4b","Type":"ContainerDied","Data":"55100d39f80869bad1168fdbfbc2e1ed0136ea532cd2814a2ebe4a09577c780c"} Nov 01 06:13:44.534031 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.525831 5040 scope.go:122] "RemoveContainer" containerID="e0467cec0d126bfccecd68df4cfbdcc9825d3d2627a83cdeba6c1a4955a9f0fb" Nov 01 06:13:44.534031 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.525973 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp" Nov 01 06:13:44.542923 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.542886 5040 generic.go:356] "Generic (PLEG): container finished" podID="ac11b327-8061-4422-af8d-a37b205d74cf" containerID="eb313244f18a2917805528aa97996609ad8fe94ceaaba137a498f8eb79798246" exitCode=0 Nov 01 06:13:44.543071 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.543043 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" event={"ID":"ac11b327-8061-4422-af8d-a37b205d74cf","Type":"ContainerDied","Data":"eb313244f18a2917805528aa97996609ad8fe94ceaaba137a498f8eb79798246"} Nov 01 06:13:44.543705 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.543663 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" Nov 01 06:13:44.544894 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.544873 5040 generic.go:356] "Generic (PLEG): container finished" podID="d6aab7db-e2e7-40ef-bf44-6873237c494f" containerID="864c5b50992c92f17b5c30b35e8c357b3e03c43378d7fc20601c29802627b0dd" exitCode=0 Nov 01 06:13:44.545055 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.545037 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" event={"ID":"d6aab7db-e2e7-40ef-bf44-6873237c494f","Type":"ContainerDied","Data":"864c5b50992c92f17b5c30b35e8c357b3e03c43378d7fc20601c29802627b0dd"} Nov 01 06:13:44.547040 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.547014 5040 generic.go:356] "Generic (PLEG): container finished" podID="8d7c546f-f34b-4824-b012-962a5e7f68c6" containerID="25047227c2789d294c9a09cb93001f4ad4745caf8d974ed0730b686e2f4d5a41" exitCode=0 Nov 01 06:13:44.547154 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.547106 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" event={"ID":"8d7c546f-f34b-4824-b012-962a5e7f68c6","Type":"ContainerDied","Data":"25047227c2789d294c9a09cb93001f4ad4745caf8d974ed0730b686e2f4d5a41"} Nov 01 06:13:44.549202 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.549176 5040 generic.go:356] "Generic (PLEG): container finished" podID="eba373b4-ae6e-45af-9833-766ae34813d5" containerID="6a77c1d0d9ff606c23eee591e1ea26ecf32e7f94bb9037b135d5d8072683b86b" exitCode=0 Nov 01 06:13:44.549289 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.549244 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" event={"ID":"eba373b4-ae6e-45af-9833-766ae34813d5","Type":"ContainerDied","Data":"6a77c1d0d9ff606c23eee591e1ea26ecf32e7f94bb9037b135d5d8072683b86b"} Nov 01 06:13:44.552202 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.552182 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" Nov 01 06:13:44.553863 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.553843 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxvx6\" (UniqueName: \"kubernetes.io/projected/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-kube-api-access-kxvx6\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.553927 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.553865 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.553927 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.553876 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.553927 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.553884 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.564826 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.564800 5040 scope.go:122] "RemoveContainer" containerID="e0467cec0d126bfccecd68df4cfbdcc9825d3d2627a83cdeba6c1a4955a9f0fb" Nov 01 06:13:44.565156 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:13:44.565124 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0467cec0d126bfccecd68df4cfbdcc9825d3d2627a83cdeba6c1a4955a9f0fb\": container with ID starting with e0467cec0d126bfccecd68df4cfbdcc9825d3d2627a83cdeba6c1a4955a9f0fb not found: ID does not exist" containerID="e0467cec0d126bfccecd68df4cfbdcc9825d3d2627a83cdeba6c1a4955a9f0fb" Nov 01 06:13:44.565261 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.565161 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0467cec0d126bfccecd68df4cfbdcc9825d3d2627a83cdeba6c1a4955a9f0fb"} err="failed to get container status \"e0467cec0d126bfccecd68df4cfbdcc9825d3d2627a83cdeba6c1a4955a9f0fb\": rpc error: code = NotFound desc = could not find container \"e0467cec0d126bfccecd68df4cfbdcc9825d3d2627a83cdeba6c1a4955a9f0fb\": container with ID starting with e0467cec0d126bfccecd68df4cfbdcc9825d3d2627a83cdeba6c1a4955a9f0fb not found: ID does not exist" Nov 01 06:13:44.567251 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.567230 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" Nov 01 06:13:44.573168 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.573146 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" Nov 01 06:13:44.654738 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.654706 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac11b327-8061-4422-af8d-a37b205d74cf-istio-token\" (UniqueName: \"kubernetes.io/projected/ac11b327-8061-4422-af8d-a37b205d74cf-istio-token\") pod \"ac11b327-8061-4422-af8d-a37b205d74cf\" (UID: \"ac11b327-8061-4422-af8d-a37b205d74cf\") " Nov 01 06:13:44.654925 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.654748 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6aab7db-e2e7-40ef-bf44-6873237c494f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6aab7db-e2e7-40ef-bf44-6873237c494f-istio-envoy\") pod \"d6aab7db-e2e7-40ef-bf44-6873237c494f\" (UID: \"d6aab7db-e2e7-40ef-bf44-6873237c494f\") " Nov 01 06:13:44.654925 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.654785 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac11b327-8061-4422-af8d-a37b205d74cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac11b327-8061-4422-af8d-a37b205d74cf-istio-podinfo\") pod \"ac11b327-8061-4422-af8d-a37b205d74cf\" (UID: \"ac11b327-8061-4422-af8d-a37b205d74cf\") " Nov 01 06:13:44.654925 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.654835 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eba373b4-ae6e-45af-9833-766ae34813d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba373b4-ae6e-45af-9833-766ae34813d5-istio-podinfo\") pod \"eba373b4-ae6e-45af-9833-766ae34813d5\" (UID: \"eba373b4-ae6e-45af-9833-766ae34813d5\") " Nov 01 06:13:44.654925 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.654873 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac11b327-8061-4422-af8d-a37b205d74cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac11b327-8061-4422-af8d-a37b205d74cf-credential-socket\") pod \"ac11b327-8061-4422-af8d-a37b205d74cf\" (UID: \"ac11b327-8061-4422-af8d-a37b205d74cf\") " Nov 01 06:13:44.654925 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.654911 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba373b4-ae6e-45af-9833-766ae34813d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba373b4-ae6e-45af-9833-766ae34813d5-istio-envoy\") pod \"eba373b4-ae6e-45af-9833-766ae34813d5\" (UID: \"eba373b4-ae6e-45af-9833-766ae34813d5\") " Nov 01 06:13:44.655176 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.654943 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba373b4-ae6e-45af-9833-766ae34813d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba373b4-ae6e-45af-9833-766ae34813d5-credential-socket\") pod \"eba373b4-ae6e-45af-9833-766ae34813d5\" (UID: \"eba373b4-ae6e-45af-9833-766ae34813d5\") " Nov 01 06:13:44.655176 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.654977 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eba373b4-ae6e-45af-9833-766ae34813d5-kube-api-access-dztvz\" (UniqueName: \"kubernetes.io/projected/eba373b4-ae6e-45af-9833-766ae34813d5-kube-api-access-dztvz\") pod \"eba373b4-ae6e-45af-9833-766ae34813d5\" (UID: \"eba373b4-ae6e-45af-9833-766ae34813d5\") " Nov 01 06:13:44.655176 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.655014 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac11b327-8061-4422-af8d-a37b205d74cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac11b327-8061-4422-af8d-a37b205d74cf-workload-certs\") pod \"ac11b327-8061-4422-af8d-a37b205d74cf\" (UID: \"ac11b327-8061-4422-af8d-a37b205d74cf\") " Nov 01 06:13:44.655176 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.655063 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7c546f-f34b-4824-b012-962a5e7f68c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7c546f-f34b-4824-b012-962a5e7f68c6-credential-socket\") pod \"8d7c546f-f34b-4824-b012-962a5e7f68c6\" (UID: \"8d7c546f-f34b-4824-b012-962a5e7f68c6\") " Nov 01 06:13:44.655176 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.655109 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eba373b4-ae6e-45af-9833-766ae34813d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba373b4-ae6e-45af-9833-766ae34813d5-istiod-ca-cert\") pod \"eba373b4-ae6e-45af-9833-766ae34813d5\" (UID: \"eba373b4-ae6e-45af-9833-766ae34813d5\") " Nov 01 06:13:44.655176 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.655140 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6aab7db-e2e7-40ef-bf44-6873237c494f-istio-token\" (UniqueName: \"kubernetes.io/projected/d6aab7db-e2e7-40ef-bf44-6873237c494f-istio-token\") pod \"d6aab7db-e2e7-40ef-bf44-6873237c494f\" (UID: \"d6aab7db-e2e7-40ef-bf44-6873237c494f\") " Nov 01 06:13:44.655470 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.655185 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d7c546f-f34b-4824-b012-962a5e7f68c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d7c546f-f34b-4824-b012-962a5e7f68c6-istio-podinfo\") pod \"8d7c546f-f34b-4824-b012-962a5e7f68c6\" (UID: \"8d7c546f-f34b-4824-b012-962a5e7f68c6\") " Nov 01 06:13:44.655470 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.655211 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba373b4-ae6e-45af-9833-766ae34813d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba373b4-ae6e-45af-9833-766ae34813d5-istio-data\") pod \"eba373b4-ae6e-45af-9833-766ae34813d5\" (UID: \"eba373b4-ae6e-45af-9833-766ae34813d5\") " Nov 01 06:13:44.655470 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.655266 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6aab7db-e2e7-40ef-bf44-6873237c494f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6aab7db-e2e7-40ef-bf44-6873237c494f-workload-certs\") pod \"d6aab7db-e2e7-40ef-bf44-6873237c494f\" (UID: \"d6aab7db-e2e7-40ef-bf44-6873237c494f\") " Nov 01 06:13:44.655470 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.655262 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac11b327-8061-4422-af8d-a37b205d74cf-credential-socket" pod "ac11b327-8061-4422-af8d-a37b205d74cf" (UID: "ac11b327-8061-4422-af8d-a37b205d74cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:13:44.655470 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.655303 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac11b327-8061-4422-af8d-a37b205d74cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac11b327-8061-4422-af8d-a37b205d74cf-workload-socket\") pod \"ac11b327-8061-4422-af8d-a37b205d74cf\" (UID: \"ac11b327-8061-4422-af8d-a37b205d74cf\") " Nov 01 06:13:44.655470 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.655350 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7c546f-f34b-4824-b012-962a5e7f68c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d7c546f-f34b-4824-b012-962a5e7f68c6-workload-certs\") pod \"8d7c546f-f34b-4824-b012-962a5e7f68c6\" (UID: \"8d7c546f-f34b-4824-b012-962a5e7f68c6\") " Nov 01 06:13:44.655470 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.655398 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6aab7db-e2e7-40ef-bf44-6873237c494f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6aab7db-e2e7-40ef-bf44-6873237c494f-credential-socket\") pod \"d6aab7db-e2e7-40ef-bf44-6873237c494f\" (UID: \"d6aab7db-e2e7-40ef-bf44-6873237c494f\") " Nov 01 06:13:44.655470 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.655419 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7c546f-f34b-4824-b012-962a5e7f68c6-credential-socket" pod "8d7c546f-f34b-4824-b012-962a5e7f68c6" (UID: "8d7c546f-f34b-4824-b012-962a5e7f68c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:13:44.655470 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.655433 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6aab7db-e2e7-40ef-bf44-6873237c494f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6aab7db-e2e7-40ef-bf44-6873237c494f-istio-podinfo\") pod \"d6aab7db-e2e7-40ef-bf44-6873237c494f\" (UID: \"d6aab7db-e2e7-40ef-bf44-6873237c494f\") " Nov 01 06:13:44.655470 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.655458 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7c546f-f34b-4824-b012-962a5e7f68c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7c546f-f34b-4824-b012-962a5e7f68c6-workload-socket\") pod \"8d7c546f-f34b-4824-b012-962a5e7f68c6\" (UID: \"8d7c546f-f34b-4824-b012-962a5e7f68c6\") " Nov 01 06:13:44.656006 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.655520 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d7c546f-f34b-4824-b012-962a5e7f68c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d7c546f-f34b-4824-b012-962a5e7f68c6-istiod-ca-cert\") pod \"8d7c546f-f34b-4824-b012-962a5e7f68c6\" (UID: \"8d7c546f-f34b-4824-b012-962a5e7f68c6\") " Nov 01 06:13:44.656006 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.655556 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d7c546f-f34b-4824-b012-962a5e7f68c6-istio-token\" (UniqueName: \"kubernetes.io/projected/8d7c546f-f34b-4824-b012-962a5e7f68c6-istio-token\") pod \"8d7c546f-f34b-4824-b012-962a5e7f68c6\" (UID: \"8d7c546f-f34b-4824-b012-962a5e7f68c6\") " Nov 01 06:13:44.656006 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.655594 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba373b4-ae6e-45af-9833-766ae34813d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba373b4-ae6e-45af-9833-766ae34813d5-workload-certs\") pod \"eba373b4-ae6e-45af-9833-766ae34813d5\" (UID: \"eba373b4-ae6e-45af-9833-766ae34813d5\") " Nov 01 06:13:44.656006 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.655627 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7c546f-f34b-4824-b012-962a5e7f68c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d7c546f-f34b-4824-b012-962a5e7f68c6-istio-data\") pod \"8d7c546f-f34b-4824-b012-962a5e7f68c6\" (UID: \"8d7c546f-f34b-4824-b012-962a5e7f68c6\") " Nov 01 06:13:44.656006 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.655660 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6aab7db-e2e7-40ef-bf44-6873237c494f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6aab7db-e2e7-40ef-bf44-6873237c494f-istio-data\") pod \"d6aab7db-e2e7-40ef-bf44-6873237c494f\" (UID: \"d6aab7db-e2e7-40ef-bf44-6873237c494f\") " Nov 01 06:13:44.656006 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.655693 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac11b327-8061-4422-af8d-a37b205d74cf-kube-api-access-6fbkq\" (UniqueName: \"kubernetes.io/projected/ac11b327-8061-4422-af8d-a37b205d74cf-kube-api-access-6fbkq\") pod \"ac11b327-8061-4422-af8d-a37b205d74cf\" (UID: \"ac11b327-8061-4422-af8d-a37b205d74cf\") " Nov 01 06:13:44.656006 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.655695 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac11b327-8061-4422-af8d-a37b205d74cf-workload-certs" pod "ac11b327-8061-4422-af8d-a37b205d74cf" (UID: "ac11b327-8061-4422-af8d-a37b205d74cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:13:44.656006 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.655730 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6aab7db-e2e7-40ef-bf44-6873237c494f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6aab7db-e2e7-40ef-bf44-6873237c494f-workload-socket\") pod \"d6aab7db-e2e7-40ef-bf44-6873237c494f\" (UID: \"d6aab7db-e2e7-40ef-bf44-6873237c494f\") " Nov 01 06:13:44.656006 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.655760 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eba373b4-ae6e-45af-9833-766ae34813d5-istio-token\" (UniqueName: \"kubernetes.io/projected/eba373b4-ae6e-45af-9833-766ae34813d5-istio-token\") pod \"eba373b4-ae6e-45af-9833-766ae34813d5\" (UID: \"eba373b4-ae6e-45af-9833-766ae34813d5\") " Nov 01 06:13:44.656006 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.655788 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba373b4-ae6e-45af-9833-766ae34813d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba373b4-ae6e-45af-9833-766ae34813d5-workload-socket\") pod \"eba373b4-ae6e-45af-9833-766ae34813d5\" (UID: \"eba373b4-ae6e-45af-9833-766ae34813d5\") " Nov 01 06:13:44.656006 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.655827 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d7c546f-f34b-4824-b012-962a5e7f68c6-kube-api-access-5xtfq\" (UniqueName: \"kubernetes.io/projected/8d7c546f-f34b-4824-b012-962a5e7f68c6-kube-api-access-5xtfq\") pod \"8d7c546f-f34b-4824-b012-962a5e7f68c6\" (UID: \"8d7c546f-f34b-4824-b012-962a5e7f68c6\") " Nov 01 06:13:44.656006 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.655857 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac11b327-8061-4422-af8d-a37b205d74cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac11b327-8061-4422-af8d-a37b205d74cf-istiod-ca-cert\") pod \"ac11b327-8061-4422-af8d-a37b205d74cf\" (UID: \"ac11b327-8061-4422-af8d-a37b205d74cf\") " Nov 01 06:13:44.656006 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.655894 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6aab7db-e2e7-40ef-bf44-6873237c494f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6aab7db-e2e7-40ef-bf44-6873237c494f-istiod-ca-cert\") pod \"d6aab7db-e2e7-40ef-bf44-6873237c494f\" (UID: \"d6aab7db-e2e7-40ef-bf44-6873237c494f\") " Nov 01 06:13:44.656006 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.655921 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba373b4-ae6e-45af-9833-766ae34813d5-credential-socket" pod "eba373b4-ae6e-45af-9833-766ae34813d5" (UID: "eba373b4-ae6e-45af-9833-766ae34813d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:13:44.656006 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.655963 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7c546f-f34b-4824-b012-962a5e7f68c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d7c546f-f34b-4824-b012-962a5e7f68c6-istio-envoy\") pod \"8d7c546f-f34b-4824-b012-962a5e7f68c6\" (UID: \"8d7c546f-f34b-4824-b012-962a5e7f68c6\") " Nov 01 06:13:44.656869 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.656028 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac11b327-8061-4422-af8d-a37b205d74cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac11b327-8061-4422-af8d-a37b205d74cf-istio-data\") pod \"ac11b327-8061-4422-af8d-a37b205d74cf\" (UID: \"ac11b327-8061-4422-af8d-a37b205d74cf\") " Nov 01 06:13:44.656869 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.656066 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6aab7db-e2e7-40ef-bf44-6873237c494f-kube-api-access-pm4zp\" (UniqueName: \"kubernetes.io/projected/d6aab7db-e2e7-40ef-bf44-6873237c494f-kube-api-access-pm4zp\") pod \"d6aab7db-e2e7-40ef-bf44-6873237c494f\" (UID: \"d6aab7db-e2e7-40ef-bf44-6873237c494f\") " Nov 01 06:13:44.656869 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.656106 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac11b327-8061-4422-af8d-a37b205d74cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac11b327-8061-4422-af8d-a37b205d74cf-istio-envoy\") pod \"ac11b327-8061-4422-af8d-a37b205d74cf\" (UID: \"ac11b327-8061-4422-af8d-a37b205d74cf\") " Nov 01 06:13:44.656869 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.656179 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6aab7db-e2e7-40ef-bf44-6873237c494f-workload-certs" pod "d6aab7db-e2e7-40ef-bf44-6873237c494f" (UID: "d6aab7db-e2e7-40ef-bf44-6873237c494f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:13:44.656869 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.656494 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d7c546f-f34b-4824-b012-962a5e7f68c6-istiod-ca-cert" pod "8d7c546f-f34b-4824-b012-962a5e7f68c6" (UID: "8d7c546f-f34b-4824-b012-962a5e7f68c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:13:44.656869 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.656761 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba373b4-ae6e-45af-9833-766ae34813d5-istio-data" pod "eba373b4-ae6e-45af-9833-766ae34813d5" (UID: "eba373b4-ae6e-45af-9833-766ae34813d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:13:44.656869 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.656805 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eba373b4-ae6e-45af-9833-766ae34813d5-istiod-ca-cert" pod "eba373b4-ae6e-45af-9833-766ae34813d5" (UID: "eba373b4-ae6e-45af-9833-766ae34813d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:13:44.657244 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.656881 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d7c546f-f34b-4824-b012-962a5e7f68c6-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.657244 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.656971 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac11b327-8061-4422-af8d-a37b205d74cf-workload-socket" pod "ac11b327-8061-4422-af8d-a37b205d74cf" (UID: "ac11b327-8061-4422-af8d-a37b205d74cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:13:44.657244 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.657014 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac11b327-8061-4422-af8d-a37b205d74cf-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.657244 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.657036 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba373b4-ae6e-45af-9833-766ae34813d5-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.657244 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.657052 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac11b327-8061-4422-af8d-a37b205d74cf-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.657244 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.657078 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7c546f-f34b-4824-b012-962a5e7f68c6-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.657244 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.657093 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba373b4-ae6e-45af-9833-766ae34813d5-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.657244 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.657099 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6aab7db-e2e7-40ef-bf44-6873237c494f-credential-socket" pod "d6aab7db-e2e7-40ef-bf44-6873237c494f" (UID: "d6aab7db-e2e7-40ef-bf44-6873237c494f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:13:44.657244 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.657108 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6aab7db-e2e7-40ef-bf44-6873237c494f-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.657675 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.657384 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7c546f-f34b-4824-b012-962a5e7f68c6-workload-certs" pod "8d7c546f-f34b-4824-b012-962a5e7f68c6" (UID: "8d7c546f-f34b-4824-b012-962a5e7f68c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:13:44.657675 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.657408 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7c546f-f34b-4824-b012-962a5e7f68c6-istio-data" pod "8d7c546f-f34b-4824-b012-962a5e7f68c6" (UID: "8d7c546f-f34b-4824-b012-962a5e7f68c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:13:44.657775 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.657718 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba373b4-ae6e-45af-9833-766ae34813d5-workload-socket" pod "eba373b4-ae6e-45af-9833-766ae34813d5" (UID: "eba373b4-ae6e-45af-9833-766ae34813d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:13:44.658129 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.658076 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7c546f-f34b-4824-b012-962a5e7f68c6-workload-socket" pod "8d7c546f-f34b-4824-b012-962a5e7f68c6" (UID: "8d7c546f-f34b-4824-b012-962a5e7f68c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:13:44.658317 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.658205 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba373b4-ae6e-45af-9833-766ae34813d5-workload-certs" pod "eba373b4-ae6e-45af-9833-766ae34813d5" (UID: "eba373b4-ae6e-45af-9833-766ae34813d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:13:44.659535 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.658975 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba373b4-ae6e-45af-9833-766ae34813d5-istio-envoy" pod "eba373b4-ae6e-45af-9833-766ae34813d5" (UID: "eba373b4-ae6e-45af-9833-766ae34813d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:13:44.659535 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.659219 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6aab7db-e2e7-40ef-bf44-6873237c494f-workload-socket" pod "d6aab7db-e2e7-40ef-bf44-6873237c494f" (UID: "d6aab7db-e2e7-40ef-bf44-6873237c494f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:13:44.659535 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.659238 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac11b327-8061-4422-af8d-a37b205d74cf-istio-data" pod "ac11b327-8061-4422-af8d-a37b205d74cf" (UID: "ac11b327-8061-4422-af8d-a37b205d74cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:13:44.659535 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.659469 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6aab7db-e2e7-40ef-bf44-6873237c494f-istio-data" pod "d6aab7db-e2e7-40ef-bf44-6873237c494f" (UID: "d6aab7db-e2e7-40ef-bf44-6873237c494f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:13:44.659535 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.659497 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac11b327-8061-4422-af8d-a37b205d74cf-istiod-ca-cert" pod "ac11b327-8061-4422-af8d-a37b205d74cf" (UID: "ac11b327-8061-4422-af8d-a37b205d74cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:13:44.659891 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.659570 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6aab7db-e2e7-40ef-bf44-6873237c494f-istiod-ca-cert" pod "d6aab7db-e2e7-40ef-bf44-6873237c494f" (UID: "d6aab7db-e2e7-40ef-bf44-6873237c494f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:13:44.660765 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.660726 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6aab7db-e2e7-40ef-bf44-6873237c494f-istio-envoy" pod "d6aab7db-e2e7-40ef-bf44-6873237c494f" (UID: "d6aab7db-e2e7-40ef-bf44-6873237c494f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:13:44.660765 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.660723 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac11b327-8061-4422-af8d-a37b205d74cf-istio-envoy" pod "ac11b327-8061-4422-af8d-a37b205d74cf" (UID: "ac11b327-8061-4422-af8d-a37b205d74cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:13:44.660896 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.660761 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eba373b4-ae6e-45af-9833-766ae34813d5-kube-api-access-dztvz" pod "eba373b4-ae6e-45af-9833-766ae34813d5" (UID: "eba373b4-ae6e-45af-9833-766ae34813d5"). InnerVolumeSpecName "kube-api-access-dztvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:13:44.660896 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.660797 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eba373b4-ae6e-45af-9833-766ae34813d5-istio-podinfo" pod "eba373b4-ae6e-45af-9833-766ae34813d5" (UID: "eba373b4-ae6e-45af-9833-766ae34813d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:13:44.660896 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.660803 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d7c546f-f34b-4824-b012-962a5e7f68c6-istio-podinfo" pod "8d7c546f-f34b-4824-b012-962a5e7f68c6" (UID: "8d7c546f-f34b-4824-b012-962a5e7f68c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:13:44.661014 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.660930 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eba373b4-ae6e-45af-9833-766ae34813d5-istio-token" pod "eba373b4-ae6e-45af-9833-766ae34813d5" (UID: "eba373b4-ae6e-45af-9833-766ae34813d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:13:44.661098 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.661081 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7c546f-f34b-4824-b012-962a5e7f68c6-istio-envoy" pod "8d7c546f-f34b-4824-b012-962a5e7f68c6" (UID: "8d7c546f-f34b-4824-b012-962a5e7f68c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:13:44.661544 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.661526 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6aab7db-e2e7-40ef-bf44-6873237c494f-kube-api-access-pm4zp" pod "d6aab7db-e2e7-40ef-bf44-6873237c494f" (UID: "d6aab7db-e2e7-40ef-bf44-6873237c494f"). InnerVolumeSpecName "kube-api-access-pm4zp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:13:44.662137 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.662117 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac11b327-8061-4422-af8d-a37b205d74cf-istio-podinfo" pod "ac11b327-8061-4422-af8d-a37b205d74cf" (UID: "ac11b327-8061-4422-af8d-a37b205d74cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:13:44.662248 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.662229 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac11b327-8061-4422-af8d-a37b205d74cf-istio-token" pod "ac11b327-8061-4422-af8d-a37b205d74cf" (UID: "ac11b327-8061-4422-af8d-a37b205d74cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:13:44.662444 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.662427 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6aab7db-e2e7-40ef-bf44-6873237c494f-istio-podinfo" pod "d6aab7db-e2e7-40ef-bf44-6873237c494f" (UID: "d6aab7db-e2e7-40ef-bf44-6873237c494f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:13:44.662603 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.662514 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d7c546f-f34b-4824-b012-962a5e7f68c6-istio-token" pod "8d7c546f-f34b-4824-b012-962a5e7f68c6" (UID: "8d7c546f-f34b-4824-b012-962a5e7f68c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:13:44.662603 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.662541 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6aab7db-e2e7-40ef-bf44-6873237c494f-istio-token" pod "d6aab7db-e2e7-40ef-bf44-6873237c494f" (UID: "d6aab7db-e2e7-40ef-bf44-6873237c494f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:13:44.662692 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.662620 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d7c546f-f34b-4824-b012-962a5e7f68c6-kube-api-access-5xtfq" pod "8d7c546f-f34b-4824-b012-962a5e7f68c6" (UID: "8d7c546f-f34b-4824-b012-962a5e7f68c6"). InnerVolumeSpecName "kube-api-access-5xtfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:13:44.663812 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.663796 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac11b327-8061-4422-af8d-a37b205d74cf-kube-api-access-6fbkq" pod "ac11b327-8061-4422-af8d-a37b205d74cf" (UID: "ac11b327-8061-4422-af8d-a37b205d74cf"). InnerVolumeSpecName "kube-api-access-6fbkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:13:44.726029 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.725995 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp"] Nov 01 06:13:44.758190 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.758158 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac11b327-8061-4422-af8d-a37b205d74cf-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.758190 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.758185 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6aab7db-e2e7-40ef-bf44-6873237c494f-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.758190 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.758196 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac11b327-8061-4422-af8d-a37b205d74cf-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.758419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.758206 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba373b4-ae6e-45af-9833-766ae34813d5-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.758419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.758214 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba373b4-ae6e-45af-9833-766ae34813d5-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.758419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.758223 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dztvz\" (UniqueName: \"kubernetes.io/projected/eba373b4-ae6e-45af-9833-766ae34813d5-kube-api-access-dztvz\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.758419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.758232 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba373b4-ae6e-45af-9833-766ae34813d5-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.758419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.758241 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6aab7db-e2e7-40ef-bf44-6873237c494f-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.758419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.758249 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d7c546f-f34b-4824-b012-962a5e7f68c6-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.758419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.758258 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac11b327-8061-4422-af8d-a37b205d74cf-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.758419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.758266 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d7c546f-f34b-4824-b012-962a5e7f68c6-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.758419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.758277 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6aab7db-e2e7-40ef-bf44-6873237c494f-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.758419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.758293 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6aab7db-e2e7-40ef-bf44-6873237c494f-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.758419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.758302 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7c546f-f34b-4824-b012-962a5e7f68c6-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.758419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.758310 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d7c546f-f34b-4824-b012-962a5e7f68c6-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.758419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.758319 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba373b4-ae6e-45af-9833-766ae34813d5-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.758419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.758327 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d7c546f-f34b-4824-b012-962a5e7f68c6-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.758419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.758335 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6aab7db-e2e7-40ef-bf44-6873237c494f-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.758419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.758343 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fbkq\" (UniqueName: \"kubernetes.io/projected/ac11b327-8061-4422-af8d-a37b205d74cf-kube-api-access-6fbkq\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.758419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.758351 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6aab7db-e2e7-40ef-bf44-6873237c494f-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.758419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.758359 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eba373b4-ae6e-45af-9833-766ae34813d5-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.758419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.758367 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba373b4-ae6e-45af-9833-766ae34813d5-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.758419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.758377 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xtfq\" (UniqueName: \"kubernetes.io/projected/8d7c546f-f34b-4824-b012-962a5e7f68c6-kube-api-access-5xtfq\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.758419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.758387 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac11b327-8061-4422-af8d-a37b205d74cf-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.758419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.758396 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6aab7db-e2e7-40ef-bf44-6873237c494f-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.758419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.758404 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d7c546f-f34b-4824-b012-962a5e7f68c6-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.758419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.758412 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac11b327-8061-4422-af8d-a37b205d74cf-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.758419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.758420 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm4zp\" (UniqueName: \"kubernetes.io/projected/d6aab7db-e2e7-40ef-bf44-6873237c494f-kube-api-access-pm4zp\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.758419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.758429 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac11b327-8061-4422-af8d-a37b205d74cf-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:13:44.760187 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:44.760165 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-prxvp"] Nov 01 06:13:45.517160 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:45.517119 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bf77172-40a2-4853-9e6a-02a0ed2cbc4b" path="/var/lib/kubelet/pods/8bf77172-40a2-4853-9e6a-02a0ed2cbc4b/volumes" Nov 01 06:13:45.557642 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:45.557605 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" event={"ID":"8d7c546f-f34b-4824-b012-962a5e7f68c6","Type":"ContainerDied","Data":"c7834f9edf9745e5639ee3f3aa8bcdf6965521c9a8603e2d960a74a38a22fbb5"} Nov 01 06:13:45.557642 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:45.557632 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs" Nov 01 06:13:45.557891 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:45.557679 5040 scope.go:122] "RemoveContainer" containerID="25047227c2789d294c9a09cb93001f4ad4745caf8d974ed0730b686e2f4d5a41" Nov 01 06:13:45.559308 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:45.559290 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" Nov 01 06:13:45.559438 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:45.559287 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6" event={"ID":"eba373b4-ae6e-45af-9833-766ae34813d5","Type":"ContainerDied","Data":"0e632d79b2b7a4ab6d92730c39a8760f977c1f593455d4d5669dbea2b192b01a"} Nov 01 06:13:45.562542 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:45.562516 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" event={"ID":"ac11b327-8061-4422-af8d-a37b205d74cf","Type":"ContainerDied","Data":"6b39865837b8f3870753f5e77e57afb52987b91eb2ebdae28bbe56e8f795ef6f"} Nov 01 06:13:45.562672 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:45.562654 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb" Nov 01 06:13:45.564063 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:45.564032 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" event={"ID":"d6aab7db-e2e7-40ef-bf44-6873237c494f","Type":"ContainerDied","Data":"aab3c2bdc213d94b68459c4f1462c5daabc2af9deb8d002a4fc7afa94278f989"} Nov 01 06:13:45.564184 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:45.564039 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z" Nov 01 06:13:45.569215 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:45.569186 5040 scope.go:122] "RemoveContainer" containerID="6a77c1d0d9ff606c23eee591e1ea26ecf32e7f94bb9037b135d5d8072683b86b" Nov 01 06:13:45.578708 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:45.578689 5040 scope.go:122] "RemoveContainer" containerID="eb313244f18a2917805528aa97996609ad8fe94ceaaba137a498f8eb79798246" Nov 01 06:13:45.590768 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:45.590741 5040 scope.go:122] "RemoveContainer" containerID="864c5b50992c92f17b5c30b35e8c357b3e03c43378d7fc20601c29802627b0dd" Nov 01 06:13:45.724414 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:45.724376 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs"] Nov 01 06:13:45.728741 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:45.728709 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-gmfjs"] Nov 01 06:13:45.839075 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:45.838988 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb"] Nov 01 06:13:45.844678 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:45.844648 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-zlvqb"] Nov 01 06:13:45.954239 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:45.954206 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6"] Nov 01 06:13:45.957458 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:45.957427 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-n7qk6"] Nov 01 06:13:46.068522 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:46.068467 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z"] Nov 01 06:13:46.073316 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:46.073285 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-djp2z"] Nov 01 06:13:47.514772 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:47.514737 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d7c546f-f34b-4824-b012-962a5e7f68c6" path="/var/lib/kubelet/pods/8d7c546f-f34b-4824-b012-962a5e7f68c6/volumes" Nov 01 06:13:47.515215 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:47.515202 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac11b327-8061-4422-af8d-a37b205d74cf" path="/var/lib/kubelet/pods/ac11b327-8061-4422-af8d-a37b205d74cf/volumes" Nov 01 06:13:47.515656 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:47.515640 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6aab7db-e2e7-40ef-bf44-6873237c494f" path="/var/lib/kubelet/pods/d6aab7db-e2e7-40ef-bf44-6873237c494f/volumes" Nov 01 06:13:47.516411 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:47.516389 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eba373b4-ae6e-45af-9833-766ae34813d5" path="/var/lib/kubelet/pods/eba373b4-ae6e-45af-9833-766ae34813d5/volumes" Nov 01 06:13:54.057965 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.057931 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95"] Nov 01 06:13:54.058404 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.058389 5040 cpu_manager.go:414] "containerMap: removing container" podUID="d6aab7db-e2e7-40ef-bf44-6873237c494f" containerName="istio-proxy" Nov 01 06:13:54.058447 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.058405 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6aab7db-e2e7-40ef-bf44-6873237c494f" containerName="istio-proxy" Nov 01 06:13:54.058447 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.058417 5040 cpu_manager.go:414] "containerMap: removing container" podUID="ac11b327-8061-4422-af8d-a37b205d74cf" containerName="istio-proxy" Nov 01 06:13:54.058447 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.058422 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac11b327-8061-4422-af8d-a37b205d74cf" containerName="istio-proxy" Nov 01 06:13:54.058447 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.058431 5040 cpu_manager.go:414] "containerMap: removing container" podUID="8bf77172-40a2-4853-9e6a-02a0ed2cbc4b" containerName="istio-proxy" Nov 01 06:13:54.058447 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.058436 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bf77172-40a2-4853-9e6a-02a0ed2cbc4b" containerName="istio-proxy" Nov 01 06:13:54.058628 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.058457 5040 cpu_manager.go:414] "containerMap: removing container" podUID="8d7c546f-f34b-4824-b012-962a5e7f68c6" containerName="istio-proxy" Nov 01 06:13:54.058628 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.058462 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d7c546f-f34b-4824-b012-962a5e7f68c6" containerName="istio-proxy" Nov 01 06:13:54.058628 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.058500 5040 cpu_manager.go:414] "containerMap: removing container" podUID="eba373b4-ae6e-45af-9833-766ae34813d5" containerName="istio-proxy" Nov 01 06:13:54.058628 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.058507 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eba373b4-ae6e-45af-9833-766ae34813d5" containerName="istio-proxy" Nov 01 06:13:54.058628 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.058590 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="eba373b4-ae6e-45af-9833-766ae34813d5" containerName="istio-proxy" Nov 01 06:13:54.058628 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.058599 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6aab7db-e2e7-40ef-bf44-6873237c494f" containerName="istio-proxy" Nov 01 06:13:54.058628 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.058605 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac11b327-8061-4422-af8d-a37b205d74cf" containerName="istio-proxy" Nov 01 06:13:54.058628 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.058611 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bf77172-40a2-4853-9e6a-02a0ed2cbc4b" containerName="istio-proxy" Nov 01 06:13:54.058628 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.058617 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d7c546f-f34b-4824-b012-962a5e7f68c6" containerName="istio-proxy" Nov 01 06:13:54.063928 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.063908 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" Nov 01 06:13:54.117051 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.117019 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-9c8fc5-openshift-default-dockercfg-6mn84\"" Nov 01 06:13:54.169425 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.163146 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ec0c34-0221-488f-a075-7f05da1461e8-credential-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95\" (UID: \"78ec0c34-0221-488f-a075-7f05da1461e8\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" Nov 01 06:13:54.169425 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.163207 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78ec0c34-0221-488f-a075-7f05da1461e8-istio-token\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95\" (UID: \"78ec0c34-0221-488f-a075-7f05da1461e8\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" Nov 01 06:13:54.169425 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.163314 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78ec0c34-0221-488f-a075-7f05da1461e8-workload-certs\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95\" (UID: \"78ec0c34-0221-488f-a075-7f05da1461e8\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" Nov 01 06:13:54.169425 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.163367 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78ec0c34-0221-488f-a075-7f05da1461e8-istiod-ca-cert\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95\" (UID: \"78ec0c34-0221-488f-a075-7f05da1461e8\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" Nov 01 06:13:54.169425 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.163454 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78ec0c34-0221-488f-a075-7f05da1461e8-istio-data\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95\" (UID: \"78ec0c34-0221-488f-a075-7f05da1461e8\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" Nov 01 06:13:54.169425 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.163516 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78ec0c34-0221-488f-a075-7f05da1461e8-istio-podinfo\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95\" (UID: \"78ec0c34-0221-488f-a075-7f05da1461e8\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" Nov 01 06:13:54.169425 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.163553 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78ec0c34-0221-488f-a075-7f05da1461e8-istio-envoy\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95\" (UID: \"78ec0c34-0221-488f-a075-7f05da1461e8\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" Nov 01 06:13:54.169425 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.163597 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ec0c34-0221-488f-a075-7f05da1461e8-workload-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95\" (UID: \"78ec0c34-0221-488f-a075-7f05da1461e8\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" Nov 01 06:13:54.169425 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.163633 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q4w6\" (UniqueName: \"kubernetes.io/projected/78ec0c34-0221-488f-a075-7f05da1461e8-kube-api-access-2q4w6\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95\" (UID: \"78ec0c34-0221-488f-a075-7f05da1461e8\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" Nov 01 06:13:54.179223 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.179189 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95"] Nov 01 06:13:54.264601 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.264563 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78ec0c34-0221-488f-a075-7f05da1461e8-workload-certs\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95\" (UID: \"78ec0c34-0221-488f-a075-7f05da1461e8\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" Nov 01 06:13:54.264759 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.264613 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78ec0c34-0221-488f-a075-7f05da1461e8-istiod-ca-cert\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95\" (UID: \"78ec0c34-0221-488f-a075-7f05da1461e8\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" Nov 01 06:13:54.264759 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.264682 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78ec0c34-0221-488f-a075-7f05da1461e8-istio-data\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95\" (UID: \"78ec0c34-0221-488f-a075-7f05da1461e8\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" Nov 01 06:13:54.264759 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.264711 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78ec0c34-0221-488f-a075-7f05da1461e8-istio-podinfo\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95\" (UID: \"78ec0c34-0221-488f-a075-7f05da1461e8\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" Nov 01 06:13:54.264759 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.264740 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78ec0c34-0221-488f-a075-7f05da1461e8-istio-envoy\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95\" (UID: \"78ec0c34-0221-488f-a075-7f05da1461e8\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" Nov 01 06:13:54.264998 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.264769 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ec0c34-0221-488f-a075-7f05da1461e8-workload-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95\" (UID: \"78ec0c34-0221-488f-a075-7f05da1461e8\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" Nov 01 06:13:54.264998 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.264798 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q4w6\" (UniqueName: \"kubernetes.io/projected/78ec0c34-0221-488f-a075-7f05da1461e8-kube-api-access-2q4w6\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95\" (UID: \"78ec0c34-0221-488f-a075-7f05da1461e8\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" Nov 01 06:13:54.264998 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.264875 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ec0c34-0221-488f-a075-7f05da1461e8-credential-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95\" (UID: \"78ec0c34-0221-488f-a075-7f05da1461e8\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" Nov 01 06:13:54.264998 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.264899 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78ec0c34-0221-488f-a075-7f05da1461e8-istio-token\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95\" (UID: \"78ec0c34-0221-488f-a075-7f05da1461e8\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" Nov 01 06:13:54.265213 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.265127 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78ec0c34-0221-488f-a075-7f05da1461e8-istio-data\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95\" (UID: \"78ec0c34-0221-488f-a075-7f05da1461e8\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" Nov 01 06:13:54.265213 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.265154 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78ec0c34-0221-488f-a075-7f05da1461e8-workload-certs\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95\" (UID: \"78ec0c34-0221-488f-a075-7f05da1461e8\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" Nov 01 06:13:54.265305 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.265217 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ec0c34-0221-488f-a075-7f05da1461e8-workload-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95\" (UID: \"78ec0c34-0221-488f-a075-7f05da1461e8\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" Nov 01 06:13:54.265717 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.265688 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ec0c34-0221-488f-a075-7f05da1461e8-credential-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95\" (UID: \"78ec0c34-0221-488f-a075-7f05da1461e8\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" Nov 01 06:13:54.265892 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.265791 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78ec0c34-0221-488f-a075-7f05da1461e8-istiod-ca-cert\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95\" (UID: \"78ec0c34-0221-488f-a075-7f05da1461e8\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" Nov 01 06:13:54.267106 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.267084 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78ec0c34-0221-488f-a075-7f05da1461e8-istio-envoy\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95\" (UID: \"78ec0c34-0221-488f-a075-7f05da1461e8\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" Nov 01 06:13:54.267763 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.267739 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78ec0c34-0221-488f-a075-7f05da1461e8-istio-podinfo\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95\" (UID: \"78ec0c34-0221-488f-a075-7f05da1461e8\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" Nov 01 06:13:54.326967 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.326883 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78ec0c34-0221-488f-a075-7f05da1461e8-istio-token\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95\" (UID: \"78ec0c34-0221-488f-a075-7f05da1461e8\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" Nov 01 06:13:54.327134 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.326971 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q4w6\" (UniqueName: \"kubernetes.io/projected/78ec0c34-0221-488f-a075-7f05da1461e8-kube-api-access-2q4w6\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95\" (UID: \"78ec0c34-0221-488f-a075-7f05da1461e8\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" Nov 01 06:13:54.378687 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.378650 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" Nov 01 06:13:54.829502 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.828113 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95"] Nov 01 06:13:54.841655 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.840923 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:13:54.841655 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.841052 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:13:54.841655 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:54.841095 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:13:55.633958 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:55.633921 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" event={"ID":"78ec0c34-0221-488f-a075-7f05da1461e8","Type":"ContainerStarted","Data":"368e949ce48e01f0526a701b6c69b201b9f8d84fc33868256bec025a8d11bc4b"} Nov 01 06:13:55.634345 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:55.633980 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" event={"ID":"78ec0c34-0221-488f-a075-7f05da1461e8","Type":"ContainerStarted","Data":"004697bf623c39b879aa28a393e582eb472984cf513d296a729f7622e47a978f"} Nov 01 06:13:55.757559 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:55.757496 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" podStartSLOduration=1.757465764 podStartE2EDuration="1.757465764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:13:55.756028614 +0000 UTC m=+1714.411145783" watchObservedRunningTime="2026-11-01 06:13:55.757465764 +0000 UTC m=+1714.412582930" Nov 01 06:13:56.378897 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:56.378858 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" Nov 01 06:13:56.380153 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:56.380123 5040 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 01 06:13:56.380303 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:56.380179 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" podUID="78ec0c34-0221-488f-a075-7f05da1461e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 01 06:13:57.379413 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:57.379352 5040 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 01 06:13:57.379898 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:57.379411 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" podUID="78ec0c34-0221-488f-a075-7f05da1461e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 01 06:13:58.379904 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:58.379869 5040 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 01 06:13:58.380411 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:58.379924 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" podUID="78ec0c34-0221-488f-a075-7f05da1461e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 01 06:13:59.379547 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:59.379469 5040 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 01 06:13:59.379760 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:13:59.379570 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" podUID="78ec0c34-0221-488f-a075-7f05da1461e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 01 06:14:00.379520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:00.379460 5040 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 01 06:14:00.380022 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:00.379566 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" podUID="78ec0c34-0221-488f-a075-7f05da1461e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 01 06:14:01.379171 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:01.379133 5040 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 01 06:14:01.379383 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:01.379192 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" podUID="78ec0c34-0221-488f-a075-7f05da1461e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 01 06:14:02.383878 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:02.383843 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" Nov 01 06:14:02.384281 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:02.384202 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" Nov 01 06:14:02.384870 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:02.384849 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" Nov 01 06:14:06.049002 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:06.048963 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-784b1d4f-sjr6m"] Nov 01 06:14:06.099249 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:06.099215 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-784b1d4f-sjr6m"] Nov 01 06:14:07.515966 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:07.515933 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05" path="/var/lib/kubelet/pods/ffacf7ee-5961-41f5-91aa-2ce1bb6e8c05/volumes" Nov 01 06:14:08.318179 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:08.318127 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj"] Nov 01 06:14:08.318421 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:08.318395 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" podUID="bbafbb5e-f923-447d-a4ac-8a0d4ef73752" containerName="istio-proxy" containerID="cri-o://10a0e62631b7b24f0983b45edf83d24db1fd3a76ccf270267a75b565fcce06f5" gracePeriod=30 Nov 01 06:14:08.368890 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:08.368854 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx"] Nov 01 06:14:08.369125 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:08.369056 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" podUID="9b5c7304-1fa5-495f-b821-79392bbd7376" containerName="istio-proxy" containerID="cri-o://52463959b30543947a2aa048b54f207ff3db2ac2ace1d64d954f7676c2daab19" gracePeriod=30 Nov 01 06:14:08.371644 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:08.371618 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd"] Nov 01 06:14:08.371836 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:08.371811 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" podUID="e2a7e8e9-21c8-4d8b-b6a9-af407459207c" containerName="istio-proxy" containerID="cri-o://b255129a5b1694db2b8b91d0482c25d9cad8797206c6d64107e2244823faf18c" gracePeriod=30 Nov 01 06:14:08.420062 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:08.420029 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj"] Nov 01 06:14:08.420248 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:08.420226 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" podUID="313135e2-fa8e-467d-ba47-91b11869b8f8" containerName="istio-proxy" containerID="cri-o://538ada513fae35add7ab3a02b5d2b611650ce11d317fe17ebe9a20b2751d5dae" gracePeriod=30 Nov 01 06:14:09.534828 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:09.534790 5040 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 06:14:09.535261 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:09.534864 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" podUID="bbafbb5e-f923-447d-a4ac-8a0d4ef73752" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 06:14:09.973377 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:14:09.973287 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:14:10.849253 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:10.849212 5040 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 06:14:10.849729 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:10.849274 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" podUID="9b5c7304-1fa5-495f-b821-79392bbd7376" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 06:14:12.973367 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:12.973328 5040 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 06:14:12.973863 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:12.973408 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" podUID="313135e2-fa8e-467d-ba47-91b11869b8f8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 06:14:13.477904 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.477869 5040 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 01 06:14:13.478079 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.477943 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" podUID="e2a7e8e9-21c8-4d8b-b6a9-af407459207c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 01 06:14:13.644565 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.644540 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" Nov 01 06:14:13.714217 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.714170 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-workload-socket\") pod \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\" (UID: \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\") " Nov 01 06:14:13.714392 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.714302 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-istio-token\" (UniqueName: \"kubernetes.io/projected/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-istio-token\") pod \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\" (UID: \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\") " Nov 01 06:14:13.714392 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.714338 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-workload-certs\") pod \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\" (UID: \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\") " Nov 01 06:14:13.714532 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.714418 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-istio-podinfo\") pod \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\" (UID: \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\") " Nov 01 06:14:13.714532 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.714447 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-istio-data\") pod \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\" (UID: \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\") " Nov 01 06:14:13.714532 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.714498 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-kube-api-access-jpx2d\" (UniqueName: \"kubernetes.io/projected/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-kube-api-access-jpx2d\") pod \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\" (UID: \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\") " Nov 01 06:14:13.714684 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.714543 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-istio-envoy\") pod \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\" (UID: \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\") " Nov 01 06:14:13.714684 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.714589 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-credential-socket\") pod \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\" (UID: \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\") " Nov 01 06:14:13.714684 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.714656 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-istiod-ca-cert\") pod \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\" (UID: \"bbafbb5e-f923-447d-a4ac-8a0d4ef73752\") " Nov 01 06:14:13.715226 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.714539 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-workload-socket" pod "bbafbb5e-f923-447d-a4ac-8a0d4ef73752" (UID: "bbafbb5e-f923-447d-a4ac-8a0d4ef73752"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:13.715226 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.714851 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-workload-certs" pod "bbafbb5e-f923-447d-a4ac-8a0d4ef73752" (UID: "bbafbb5e-f923-447d-a4ac-8a0d4ef73752"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:13.715226 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.715123 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:13.715226 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.715145 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:13.715226 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.715196 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-istio-data" pod "bbafbb5e-f923-447d-a4ac-8a0d4ef73752" (UID: "bbafbb5e-f923-447d-a4ac-8a0d4ef73752"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:13.715915 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.715887 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-istiod-ca-cert" pod "bbafbb5e-f923-447d-a4ac-8a0d4ef73752" (UID: "bbafbb5e-f923-447d-a4ac-8a0d4ef73752"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:14:13.719551 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.716122 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-credential-socket" pod "bbafbb5e-f923-447d-a4ac-8a0d4ef73752" (UID: "bbafbb5e-f923-447d-a4ac-8a0d4ef73752"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:13.719551 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.718161 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-istio-token" pod "bbafbb5e-f923-447d-a4ac-8a0d4ef73752" (UID: "bbafbb5e-f923-447d-a4ac-8a0d4ef73752"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:14:13.719551 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.719152 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-istio-podinfo" pod "bbafbb5e-f923-447d-a4ac-8a0d4ef73752" (UID: "bbafbb5e-f923-447d-a4ac-8a0d4ef73752"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:14:13.719551 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.719156 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-kube-api-access-jpx2d" pod "bbafbb5e-f923-447d-a4ac-8a0d4ef73752" (UID: "bbafbb5e-f923-447d-a4ac-8a0d4ef73752"). InnerVolumeSpecName "kube-api-access-jpx2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:14:13.720275 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.720255 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-istio-envoy" pod "bbafbb5e-f923-447d-a4ac-8a0d4ef73752" (UID: "bbafbb5e-f923-447d-a4ac-8a0d4ef73752"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:13.759309 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.759280 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" Nov 01 06:14:13.768071 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.768046 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" Nov 01 06:14:13.771080 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.771043 5040 generic.go:356] "Generic (PLEG): container finished" podID="e2a7e8e9-21c8-4d8b-b6a9-af407459207c" containerID="b255129a5b1694db2b8b91d0482c25d9cad8797206c6d64107e2244823faf18c" exitCode=0 Nov 01 06:14:13.771199 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.771097 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" event={"ID":"e2a7e8e9-21c8-4d8b-b6a9-af407459207c","Type":"ContainerDied","Data":"b255129a5b1694db2b8b91d0482c25d9cad8797206c6d64107e2244823faf18c"} Nov 01 06:14:13.771265 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.771202 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" event={"ID":"e2a7e8e9-21c8-4d8b-b6a9-af407459207c","Type":"ContainerDied","Data":"0fced4948536555ec45e6a063deb416a6d411fa88e756e494e1cc1698985ab32"} Nov 01 06:14:13.771265 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.771232 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fced4948536555ec45e6a063deb416a6d411fa88e756e494e1cc1698985ab32" Nov 01 06:14:13.772288 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.772266 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" Nov 01 06:14:13.772503 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.772456 5040 generic.go:356] "Generic (PLEG): container finished" podID="313135e2-fa8e-467d-ba47-91b11869b8f8" containerID="538ada513fae35add7ab3a02b5d2b611650ce11d317fe17ebe9a20b2751d5dae" exitCode=0 Nov 01 06:14:13.772599 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.772497 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" event={"ID":"313135e2-fa8e-467d-ba47-91b11869b8f8","Type":"ContainerDied","Data":"538ada513fae35add7ab3a02b5d2b611650ce11d317fe17ebe9a20b2751d5dae"} Nov 01 06:14:13.772599 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.772557 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" event={"ID":"313135e2-fa8e-467d-ba47-91b11869b8f8","Type":"ContainerDied","Data":"4b1cb543e74187fae6b7bf8c0ecbaa6880f69719091e001446fcc4763575d93f"} Nov 01 06:14:13.772599 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.772593 5040 scope.go:122] "RemoveContainer" containerID="538ada513fae35add7ab3a02b5d2b611650ce11d317fe17ebe9a20b2751d5dae" Nov 01 06:14:13.774615 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.774591 5040 generic.go:356] "Generic (PLEG): container finished" podID="bbafbb5e-f923-447d-a4ac-8a0d4ef73752" containerID="10a0e62631b7b24f0983b45edf83d24db1fd3a76ccf270267a75b565fcce06f5" exitCode=0 Nov 01 06:14:13.774725 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.774640 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" Nov 01 06:14:13.774725 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.774708 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" event={"ID":"bbafbb5e-f923-447d-a4ac-8a0d4ef73752","Type":"ContainerDied","Data":"10a0e62631b7b24f0983b45edf83d24db1fd3a76ccf270267a75b565fcce06f5"} Nov 01 06:14:13.774827 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.774752 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj" event={"ID":"bbafbb5e-f923-447d-a4ac-8a0d4ef73752","Type":"ContainerDied","Data":"f7e69d2f40be441d5095504761be0b53c01f5ef13226315835f486fc49384750"} Nov 01 06:14:13.783507 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.775571 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj" Nov 01 06:14:13.783507 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.777072 5040 generic.go:356] "Generic (PLEG): container finished" podID="9b5c7304-1fa5-495f-b821-79392bbd7376" containerID="52463959b30543947a2aa048b54f207ff3db2ac2ace1d64d954f7676c2daab19" exitCode=0 Nov 01 06:14:13.783507 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.777142 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" Nov 01 06:14:13.783507 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.777190 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" event={"ID":"9b5c7304-1fa5-495f-b821-79392bbd7376","Type":"ContainerDied","Data":"52463959b30543947a2aa048b54f207ff3db2ac2ace1d64d954f7676c2daab19"} Nov 01 06:14:13.783507 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.777218 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx" event={"ID":"9b5c7304-1fa5-495f-b821-79392bbd7376","Type":"ContainerDied","Data":"862e06941475ce120ee731b6a5defbefdfbcd5fb4c26f0bc48902e6155f1b4e4"} Nov 01 06:14:13.786895 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.786875 5040 scope.go:122] "RemoveContainer" containerID="538ada513fae35add7ab3a02b5d2b611650ce11d317fe17ebe9a20b2751d5dae" Nov 01 06:14:13.787205 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:14:13.787186 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"538ada513fae35add7ab3a02b5d2b611650ce11d317fe17ebe9a20b2751d5dae\": container with ID starting with 538ada513fae35add7ab3a02b5d2b611650ce11d317fe17ebe9a20b2751d5dae not found: ID does not exist" containerID="538ada513fae35add7ab3a02b5d2b611650ce11d317fe17ebe9a20b2751d5dae" Nov 01 06:14:13.787280 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.787219 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"538ada513fae35add7ab3a02b5d2b611650ce11d317fe17ebe9a20b2751d5dae"} err="failed to get container status \"538ada513fae35add7ab3a02b5d2b611650ce11d317fe17ebe9a20b2751d5dae\": rpc error: code = NotFound desc = could not find container \"538ada513fae35add7ab3a02b5d2b611650ce11d317fe17ebe9a20b2751d5dae\": container with ID starting with 538ada513fae35add7ab3a02b5d2b611650ce11d317fe17ebe9a20b2751d5dae not found: ID does not exist" Nov 01 06:14:13.787280 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.787246 5040 scope.go:122] "RemoveContainer" containerID="10a0e62631b7b24f0983b45edf83d24db1fd3a76ccf270267a75b565fcce06f5" Nov 01 06:14:13.797143 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.797092 5040 scope.go:122] "RemoveContainer" containerID="10a0e62631b7b24f0983b45edf83d24db1fd3a76ccf270267a75b565fcce06f5" Nov 01 06:14:13.797524 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:14:13.797461 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10a0e62631b7b24f0983b45edf83d24db1fd3a76ccf270267a75b565fcce06f5\": container with ID starting with 10a0e62631b7b24f0983b45edf83d24db1fd3a76ccf270267a75b565fcce06f5 not found: ID does not exist" containerID="10a0e62631b7b24f0983b45edf83d24db1fd3a76ccf270267a75b565fcce06f5" Nov 01 06:14:13.797644 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.797534 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10a0e62631b7b24f0983b45edf83d24db1fd3a76ccf270267a75b565fcce06f5"} err="failed to get container status \"10a0e62631b7b24f0983b45edf83d24db1fd3a76ccf270267a75b565fcce06f5\": rpc error: code = NotFound desc = could not find container \"10a0e62631b7b24f0983b45edf83d24db1fd3a76ccf270267a75b565fcce06f5\": container with ID starting with 10a0e62631b7b24f0983b45edf83d24db1fd3a76ccf270267a75b565fcce06f5 not found: ID does not exist" Nov 01 06:14:13.797644 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.797558 5040 scope.go:122] "RemoveContainer" containerID="52463959b30543947a2aa048b54f207ff3db2ac2ace1d64d954f7676c2daab19" Nov 01 06:14:13.807178 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.807158 5040 scope.go:122] "RemoveContainer" containerID="52463959b30543947a2aa048b54f207ff3db2ac2ace1d64d954f7676c2daab19" Nov 01 06:14:13.807517 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:14:13.807459 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52463959b30543947a2aa048b54f207ff3db2ac2ace1d64d954f7676c2daab19\": container with ID starting with 52463959b30543947a2aa048b54f207ff3db2ac2ace1d64d954f7676c2daab19 not found: ID does not exist" containerID="52463959b30543947a2aa048b54f207ff3db2ac2ace1d64d954f7676c2daab19" Nov 01 06:14:13.807618 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.807529 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52463959b30543947a2aa048b54f207ff3db2ac2ace1d64d954f7676c2daab19"} err="failed to get container status \"52463959b30543947a2aa048b54f207ff3db2ac2ace1d64d954f7676c2daab19\": rpc error: code = NotFound desc = could not find container \"52463959b30543947a2aa048b54f207ff3db2ac2ace1d64d954f7676c2daab19\": container with ID starting with 52463959b30543947a2aa048b54f207ff3db2ac2ace1d64d954f7676c2daab19 not found: ID does not exist" Nov 01 06:14:13.815831 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.815809 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/313135e2-fa8e-467d-ba47-91b11869b8f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/313135e2-fa8e-467d-ba47-91b11869b8f8-istio-data\") pod \"313135e2-fa8e-467d-ba47-91b11869b8f8\" (UID: \"313135e2-fa8e-467d-ba47-91b11869b8f8\") " Nov 01 06:14:13.815917 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.815862 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/313135e2-fa8e-467d-ba47-91b11869b8f8-kube-api-access-9mqj7\" (UniqueName: \"kubernetes.io/projected/313135e2-fa8e-467d-ba47-91b11869b8f8-kube-api-access-9mqj7\") pod \"313135e2-fa8e-467d-ba47-91b11869b8f8\" (UID: \"313135e2-fa8e-467d-ba47-91b11869b8f8\") " Nov 01 06:14:13.815966 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.815943 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/313135e2-fa8e-467d-ba47-91b11869b8f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/313135e2-fa8e-467d-ba47-91b11869b8f8-istio-envoy\") pod \"313135e2-fa8e-467d-ba47-91b11869b8f8\" (UID: \"313135e2-fa8e-467d-ba47-91b11869b8f8\") " Nov 01 06:14:13.816025 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.815992 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/313135e2-fa8e-467d-ba47-91b11869b8f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/313135e2-fa8e-467d-ba47-91b11869b8f8-istio-podinfo\") pod \"313135e2-fa8e-467d-ba47-91b11869b8f8\" (UID: \"313135e2-fa8e-467d-ba47-91b11869b8f8\") " Nov 01 06:14:13.816025 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.816019 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/313135e2-fa8e-467d-ba47-91b11869b8f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/313135e2-fa8e-467d-ba47-91b11869b8f8-workload-certs\") pod \"313135e2-fa8e-467d-ba47-91b11869b8f8\" (UID: \"313135e2-fa8e-467d-ba47-91b11869b8f8\") " Nov 01 06:14:13.816304 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.816107 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/313135e2-fa8e-467d-ba47-91b11869b8f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/313135e2-fa8e-467d-ba47-91b11869b8f8-workload-socket\") pod \"313135e2-fa8e-467d-ba47-91b11869b8f8\" (UID: \"313135e2-fa8e-467d-ba47-91b11869b8f8\") " Nov 01 06:14:13.816304 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.816168 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313135e2-fa8e-467d-ba47-91b11869b8f8-istio-data" pod "313135e2-fa8e-467d-ba47-91b11869b8f8" (UID: "313135e2-fa8e-467d-ba47-91b11869b8f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:13.816304 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.816189 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/313135e2-fa8e-467d-ba47-91b11869b8f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/313135e2-fa8e-467d-ba47-91b11869b8f8-istiod-ca-cert\") pod \"313135e2-fa8e-467d-ba47-91b11869b8f8\" (UID: \"313135e2-fa8e-467d-ba47-91b11869b8f8\") " Nov 01 06:14:13.816304 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.816219 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/313135e2-fa8e-467d-ba47-91b11869b8f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/313135e2-fa8e-467d-ba47-91b11869b8f8-credential-socket\") pod \"313135e2-fa8e-467d-ba47-91b11869b8f8\" (UID: \"313135e2-fa8e-467d-ba47-91b11869b8f8\") " Nov 01 06:14:13.816304 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.816259 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/313135e2-fa8e-467d-ba47-91b11869b8f8-istio-token\" (UniqueName: \"kubernetes.io/projected/313135e2-fa8e-467d-ba47-91b11869b8f8-istio-token\") pod \"313135e2-fa8e-467d-ba47-91b11869b8f8\" (UID: \"313135e2-fa8e-467d-ba47-91b11869b8f8\") " Nov 01 06:14:13.816591 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.816449 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313135e2-fa8e-467d-ba47-91b11869b8f8-workload-socket" pod "313135e2-fa8e-467d-ba47-91b11869b8f8" (UID: "313135e2-fa8e-467d-ba47-91b11869b8f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:13.816591 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.816516 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313135e2-fa8e-467d-ba47-91b11869b8f8-workload-certs" pod "313135e2-fa8e-467d-ba47-91b11869b8f8" (UID: "313135e2-fa8e-467d-ba47-91b11869b8f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:13.817359 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.817131 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313135e2-fa8e-467d-ba47-91b11869b8f8-credential-socket" pod "313135e2-fa8e-467d-ba47-91b11869b8f8" (UID: "313135e2-fa8e-467d-ba47-91b11869b8f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:13.817359 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.817162 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:13.817359 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.817191 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/313135e2-fa8e-467d-ba47-91b11869b8f8-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:13.817359 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.817205 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:13.817359 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.817219 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:13.817359 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.817234 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpx2d\" (UniqueName: \"kubernetes.io/projected/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-kube-api-access-jpx2d\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:13.817359 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.817250 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:13.817359 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.817264 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/313135e2-fa8e-467d-ba47-91b11869b8f8-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:13.817359 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.817279 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:13.817359 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.817295 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbafbb5e-f923-447d-a4ac-8a0d4ef73752-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:13.817359 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.817308 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/313135e2-fa8e-467d-ba47-91b11869b8f8-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:13.817359 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.817353 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/313135e2-fa8e-467d-ba47-91b11869b8f8-istiod-ca-cert" pod "313135e2-fa8e-467d-ba47-91b11869b8f8" (UID: "313135e2-fa8e-467d-ba47-91b11869b8f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:14:13.818472 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.818443 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313135e2-fa8e-467d-ba47-91b11869b8f8-istio-envoy" pod "313135e2-fa8e-467d-ba47-91b11869b8f8" (UID: "313135e2-fa8e-467d-ba47-91b11869b8f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:13.818472 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.818451 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/313135e2-fa8e-467d-ba47-91b11869b8f8-kube-api-access-9mqj7" pod "313135e2-fa8e-467d-ba47-91b11869b8f8" (UID: "313135e2-fa8e-467d-ba47-91b11869b8f8"). InnerVolumeSpecName "kube-api-access-9mqj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:14:13.818616 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.818563 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/313135e2-fa8e-467d-ba47-91b11869b8f8-istio-podinfo" pod "313135e2-fa8e-467d-ba47-91b11869b8f8" (UID: "313135e2-fa8e-467d-ba47-91b11869b8f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:14:13.818813 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.818793 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/313135e2-fa8e-467d-ba47-91b11869b8f8-istio-token" pod "313135e2-fa8e-467d-ba47-91b11869b8f8" (UID: "313135e2-fa8e-467d-ba47-91b11869b8f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:14:13.918225 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.918139 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b5c7304-1fa5-495f-b821-79392bbd7376-kube-api-access-9k9v5\" (UniqueName: \"kubernetes.io/projected/9b5c7304-1fa5-495f-b821-79392bbd7376-kube-api-access-9k9v5\") pod \"9b5c7304-1fa5-495f-b821-79392bbd7376\" (UID: \"9b5c7304-1fa5-495f-b821-79392bbd7376\") " Nov 01 06:14:13.918225 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.918209 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-istio-data\") pod \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\" (UID: \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\") " Nov 01 06:14:13.918450 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.918244 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b5c7304-1fa5-495f-b821-79392bbd7376-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b5c7304-1fa5-495f-b821-79392bbd7376-workload-certs\") pod \"9b5c7304-1fa5-495f-b821-79392bbd7376\" (UID: \"9b5c7304-1fa5-495f-b821-79392bbd7376\") " Nov 01 06:14:13.918450 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.918277 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-workload-socket\") pod \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\" (UID: \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\") " Nov 01 06:14:13.918450 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.918313 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-kube-api-access-xwz2t\" (UniqueName: \"kubernetes.io/projected/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-kube-api-access-xwz2t\") pod \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\" (UID: \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\") " Nov 01 06:14:13.918450 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.918345 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b5c7304-1fa5-495f-b821-79392bbd7376-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5c7304-1fa5-495f-b821-79392bbd7376-workload-socket\") pod \"9b5c7304-1fa5-495f-b821-79392bbd7376\" (UID: \"9b5c7304-1fa5-495f-b821-79392bbd7376\") " Nov 01 06:14:13.918450 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.918372 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-credential-socket\") pod \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\" (UID: \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\") " Nov 01 06:14:13.918450 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.918439 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b5c7304-1fa5-495f-b821-79392bbd7376-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5c7304-1fa5-495f-b821-79392bbd7376-credential-socket\") pod \"9b5c7304-1fa5-495f-b821-79392bbd7376\" (UID: \"9b5c7304-1fa5-495f-b821-79392bbd7376\") " Nov 01 06:14:13.918828 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.918472 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-istio-podinfo\") pod \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\" (UID: \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\") " Nov 01 06:14:13.918828 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.918535 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-workload-certs\") pod \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\" (UID: \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\") " Nov 01 06:14:13.918828 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.918540 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-istio-data" pod "e2a7e8e9-21c8-4d8b-b6a9-af407459207c" (UID: "e2a7e8e9-21c8-4d8b-b6a9-af407459207c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:13.918828 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.918565 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b5c7304-1fa5-495f-b821-79392bbd7376-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b5c7304-1fa5-495f-b821-79392bbd7376-istiod-ca-cert\") pod \"9b5c7304-1fa5-495f-b821-79392bbd7376\" (UID: \"9b5c7304-1fa5-495f-b821-79392bbd7376\") " Nov 01 06:14:13.918828 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.918589 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-workload-socket" pod "e2a7e8e9-21c8-4d8b-b6a9-af407459207c" (UID: "e2a7e8e9-21c8-4d8b-b6a9-af407459207c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:13.918828 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.918594 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-istiod-ca-cert\") pod \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\" (UID: \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\") " Nov 01 06:14:13.918828 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.918646 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5c7304-1fa5-495f-b821-79392bbd7376-workload-certs" pod "9b5c7304-1fa5-495f-b821-79392bbd7376" (UID: "9b5c7304-1fa5-495f-b821-79392bbd7376"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:13.918828 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.918680 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5c7304-1fa5-495f-b821-79392bbd7376-workload-socket" pod "9b5c7304-1fa5-495f-b821-79392bbd7376" (UID: "9b5c7304-1fa5-495f-b821-79392bbd7376"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:13.918828 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.918692 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b5c7304-1fa5-495f-b821-79392bbd7376-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b5c7304-1fa5-495f-b821-79392bbd7376-istio-envoy\") pod \"9b5c7304-1fa5-495f-b821-79392bbd7376\" (UID: \"9b5c7304-1fa5-495f-b821-79392bbd7376\") " Nov 01 06:14:13.918828 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.918747 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b5c7304-1fa5-495f-b821-79392bbd7376-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b5c7304-1fa5-495f-b821-79392bbd7376-istio-data\") pod \"9b5c7304-1fa5-495f-b821-79392bbd7376\" (UID: \"9b5c7304-1fa5-495f-b821-79392bbd7376\") " Nov 01 06:14:13.918828 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.918761 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5c7304-1fa5-495f-b821-79392bbd7376-credential-socket" pod "9b5c7304-1fa5-495f-b821-79392bbd7376" (UID: "9b5c7304-1fa5-495f-b821-79392bbd7376"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:13.918828 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.918776 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b5c7304-1fa5-495f-b821-79392bbd7376-istio-token\" (UniqueName: \"kubernetes.io/projected/9b5c7304-1fa5-495f-b821-79392bbd7376-istio-token\") pod \"9b5c7304-1fa5-495f-b821-79392bbd7376\" (UID: \"9b5c7304-1fa5-495f-b821-79392bbd7376\") " Nov 01 06:14:13.918828 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.918807 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-istio-envoy\") pod \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\" (UID: \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\") " Nov 01 06:14:13.919928 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.918868 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-credential-socket" pod "e2a7e8e9-21c8-4d8b-b6a9-af407459207c" (UID: "e2a7e8e9-21c8-4d8b-b6a9-af407459207c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:13.919928 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.918882 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b5c7304-1fa5-495f-b821-79392bbd7376-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b5c7304-1fa5-495f-b821-79392bbd7376-istio-podinfo\") pod \"9b5c7304-1fa5-495f-b821-79392bbd7376\" (UID: \"9b5c7304-1fa5-495f-b821-79392bbd7376\") " Nov 01 06:14:13.919928 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.918912 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-workload-certs" pod "e2a7e8e9-21c8-4d8b-b6a9-af407459207c" (UID: "e2a7e8e9-21c8-4d8b-b6a9-af407459207c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:13.919928 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.918921 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-istiod-ca-cert" pod "e2a7e8e9-21c8-4d8b-b6a9-af407459207c" (UID: "e2a7e8e9-21c8-4d8b-b6a9-af407459207c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:14:13.919928 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.918981 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-istio-token\" (UniqueName: \"kubernetes.io/projected/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-istio-token\") pod \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\" (UID: \"e2a7e8e9-21c8-4d8b-b6a9-af407459207c\") " Nov 01 06:14:13.919928 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.919341 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b5c7304-1fa5-495f-b821-79392bbd7376-istiod-ca-cert" pod "9b5c7304-1fa5-495f-b821-79392bbd7376" (UID: "9b5c7304-1fa5-495f-b821-79392bbd7376"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:14:13.919928 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.919572 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/313135e2-fa8e-467d-ba47-91b11869b8f8-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:13.919928 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.919637 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5c7304-1fa5-495f-b821-79392bbd7376-istio-data" pod "9b5c7304-1fa5-495f-b821-79392bbd7376" (UID: "9b5c7304-1fa5-495f-b821-79392bbd7376"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:13.919928 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.919685 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/313135e2-fa8e-467d-ba47-91b11869b8f8-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:13.919928 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.919703 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/313135e2-fa8e-467d-ba47-91b11869b8f8-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:13.919928 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.919718 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:13.919928 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.919732 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b5c7304-1fa5-495f-b821-79392bbd7376-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:13.919928 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.919746 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:13.919928 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.919761 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5c7304-1fa5-495f-b821-79392bbd7376-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:13.919928 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.919776 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:13.919928 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.919793 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mqj7\" (UniqueName: \"kubernetes.io/projected/313135e2-fa8e-467d-ba47-91b11869b8f8-kube-api-access-9mqj7\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:13.919928 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.919808 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5c7304-1fa5-495f-b821-79392bbd7376-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:13.919928 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.919823 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:13.919928 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.919838 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b5c7304-1fa5-495f-b821-79392bbd7376-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:13.919928 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.919852 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:13.919928 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.919867 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/313135e2-fa8e-467d-ba47-91b11869b8f8-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:13.919928 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.919884 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/313135e2-fa8e-467d-ba47-91b11869b8f8-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:13.921212 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.921181 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b5c7304-1fa5-495f-b821-79392bbd7376-kube-api-access-9k9v5" pod "9b5c7304-1fa5-495f-b821-79392bbd7376" (UID: "9b5c7304-1fa5-495f-b821-79392bbd7376"). InnerVolumeSpecName "kube-api-access-9k9v5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:14:13.921514 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.921462 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b5c7304-1fa5-495f-b821-79392bbd7376-istio-token" pod "9b5c7304-1fa5-495f-b821-79392bbd7376" (UID: "9b5c7304-1fa5-495f-b821-79392bbd7376"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:14:13.921603 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.921544 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b5c7304-1fa5-495f-b821-79392bbd7376-istio-podinfo" pod "9b5c7304-1fa5-495f-b821-79392bbd7376" (UID: "9b5c7304-1fa5-495f-b821-79392bbd7376"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:14:13.921644 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.921596 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-kube-api-access-xwz2t" pod "e2a7e8e9-21c8-4d8b-b6a9-af407459207c" (UID: "e2a7e8e9-21c8-4d8b-b6a9-af407459207c"). InnerVolumeSpecName "kube-api-access-xwz2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:14:13.921644 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.921628 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-istio-podinfo" pod "e2a7e8e9-21c8-4d8b-b6a9-af407459207c" (UID: "e2a7e8e9-21c8-4d8b-b6a9-af407459207c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:14:13.921820 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.921789 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-istio-token" pod "e2a7e8e9-21c8-4d8b-b6a9-af407459207c" (UID: "e2a7e8e9-21c8-4d8b-b6a9-af407459207c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:14:13.921983 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.921963 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5c7304-1fa5-495f-b821-79392bbd7376-istio-envoy" pod "9b5c7304-1fa5-495f-b821-79392bbd7376" (UID: "9b5c7304-1fa5-495f-b821-79392bbd7376"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:13.922718 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:13.922703 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-istio-envoy" pod "e2a7e8e9-21c8-4d8b-b6a9-af407459207c" (UID: "e2a7e8e9-21c8-4d8b-b6a9-af407459207c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:14.020727 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:14.020687 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b5c7304-1fa5-495f-b821-79392bbd7376-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:14.020727 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:14.020724 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:14.021158 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:14.020738 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k9v5\" (UniqueName: \"kubernetes.io/projected/9b5c7304-1fa5-495f-b821-79392bbd7376-kube-api-access-9k9v5\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:14.021158 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:14.020755 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwz2t\" (UniqueName: \"kubernetes.io/projected/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-kube-api-access-xwz2t\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:14.021158 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:14.020767 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:14.021158 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:14.020778 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b5c7304-1fa5-495f-b821-79392bbd7376-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:14.021158 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:14.020791 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b5c7304-1fa5-495f-b821-79392bbd7376-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:14.021158 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:14.020802 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b5c7304-1fa5-495f-b821-79392bbd7376-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:14.021158 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:14.020815 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a7e8e9-21c8-4d8b-b6a9-af407459207c-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:14.052857 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:14.052821 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj"] Nov 01 06:14:14.056798 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:14.056767 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-898kj"] Nov 01 06:14:14.414237 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:14.414198 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj"] Nov 01 06:14:14.419442 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:14.419316 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-rgjcj"] Nov 01 06:14:14.528203 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:14.528171 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx"] Nov 01 06:14:14.533046 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:14.533018 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-7q9bx"] Nov 01 06:14:14.787086 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:14.787059 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd" Nov 01 06:14:14.957058 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:14.957020 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd"] Nov 01 06:14:14.961306 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:14.961274 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-zglwd"] Nov 01 06:14:15.514472 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:15.514439 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="313135e2-fa8e-467d-ba47-91b11869b8f8" path="/var/lib/kubelet/pods/313135e2-fa8e-467d-ba47-91b11869b8f8/volumes" Nov 01 06:14:15.514987 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:15.514966 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b5c7304-1fa5-495f-b821-79392bbd7376" path="/var/lib/kubelet/pods/9b5c7304-1fa5-495f-b821-79392bbd7376/volumes" Nov 01 06:14:15.515417 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:15.515405 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbafbb5e-f923-447d-a4ac-8a0d4ef73752" path="/var/lib/kubelet/pods/bbafbb5e-f923-447d-a4ac-8a0d4ef73752/volumes" Nov 01 06:14:15.516228 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:15.516212 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2a7e8e9-21c8-4d8b-b6a9-af407459207c" path="/var/lib/kubelet/pods/e2a7e8e9-21c8-4d8b-b6a9-af407459207c/volumes" Nov 01 06:14:22.352332 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:22.352295 5040 scope.go:122] "RemoveContainer" containerID="be8b20a6ec342f0c07a0b20bc82f256d83c3f641400c6fbcd6a10dc3ec29be53" Nov 01 06:14:22.362127 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:22.362106 5040 scope.go:122] "RemoveContainer" containerID="18d971d620c08f44fba39210a7d87318730e80c4066e3c7ebcf1d28367c964c0" Nov 01 06:14:22.371687 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:22.371665 5040 scope.go:122] "RemoveContainer" containerID="4374b0a4a8bb149f84a283cc8a5fdd7601a58d3e2fe39d7a81ab444f30bf918c" Nov 01 06:14:22.382841 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:22.380550 5040 scope.go:122] "RemoveContainer" containerID="83b8efb13e934fb4ec2e601a27f77072bec403e586ccca908018025d73d442ba" Nov 01 06:14:22.392861 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:22.392840 5040 scope.go:122] "RemoveContainer" containerID="15aa1987af6e52e5fc959721bcde3aba2bae8d7f250f69603921956dbde14b2d" Nov 01 06:14:22.401646 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:22.401587 5040 scope.go:122] "RemoveContainer" containerID="b255129a5b1694db2b8b91d0482c25d9cad8797206c6d64107e2244823faf18c" Nov 01 06:14:22.411551 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:22.411529 5040 scope.go:122] "RemoveContainer" containerID="04b4d41d2bd0187f7b8127e6904f2dad50dd7c281c0d5ae35e7d00f6b77dd70e" Nov 01 06:14:23.437261 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:23.437228 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1ac44e5-xghg9"] Nov 01 06:14:23.437678 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:23.437664 5040 cpu_manager.go:414] "containerMap: removing container" podUID="313135e2-fa8e-467d-ba47-91b11869b8f8" containerName="istio-proxy" Nov 01 06:14:23.437678 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:23.437675 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="313135e2-fa8e-467d-ba47-91b11869b8f8" containerName="istio-proxy" Nov 01 06:14:23.437763 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:23.437690 5040 cpu_manager.go:414] "containerMap: removing container" podUID="bbafbb5e-f923-447d-a4ac-8a0d4ef73752" containerName="istio-proxy" Nov 01 06:14:23.437763 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:23.437696 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbafbb5e-f923-447d-a4ac-8a0d4ef73752" containerName="istio-proxy" Nov 01 06:14:23.437763 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:23.437703 5040 cpu_manager.go:414] "containerMap: removing container" podUID="e2a7e8e9-21c8-4d8b-b6a9-af407459207c" containerName="istio-proxy" Nov 01 06:14:23.437763 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:23.437708 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2a7e8e9-21c8-4d8b-b6a9-af407459207c" containerName="istio-proxy" Nov 01 06:14:23.437763 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:23.437737 5040 cpu_manager.go:414] "containerMap: removing container" podUID="9b5c7304-1fa5-495f-b821-79392bbd7376" containerName="istio-proxy" Nov 01 06:14:23.437763 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:23.437742 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b5c7304-1fa5-495f-b821-79392bbd7376" containerName="istio-proxy" Nov 01 06:14:23.437962 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:23.437810 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbafbb5e-f923-447d-a4ac-8a0d4ef73752" containerName="istio-proxy" Nov 01 06:14:23.437962 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:23.437818 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="313135e2-fa8e-467d-ba47-91b11869b8f8" containerName="istio-proxy" Nov 01 06:14:23.437962 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:23.437825 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2a7e8e9-21c8-4d8b-b6a9-af407459207c" containerName="istio-proxy" Nov 01 06:14:23.437962 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:23.437836 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b5c7304-1fa5-495f-b821-79392bbd7376" containerName="istio-proxy" Nov 01 06:14:23.444134 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:23.444114 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ac44e5-xghg9" Nov 01 06:14:23.497394 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:23.497360 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:14:23.497697 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:23.497607 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:14:23.498102 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:23.498073 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:14:23.498228 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:23.498208 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:14:23.498830 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:23.498805 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1ac44e5-xghg9"] Nov 01 06:14:23.521589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:23.521557 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1ac44e5-xghg9\" (UID: \"6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ac44e5-xghg9" Nov 01 06:14:23.521779 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:23.521639 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlv6p\" (UniqueName: \"kubernetes.io/projected/6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff-kube-api-access-nlv6p\") pod \"maas-api-revoke-sub-c1ac44e5-xghg9\" (UID: \"6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ac44e5-xghg9" Nov 01 06:14:23.521779 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:23.521662 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1ac44e5-xghg9\" (UID: \"6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ac44e5-xghg9" Nov 01 06:14:23.623300 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:23.623269 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1ac44e5-xghg9\" (UID: \"6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ac44e5-xghg9" Nov 01 06:14:23.623473 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:23.623314 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlv6p\" (UniqueName: \"kubernetes.io/projected/6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff-kube-api-access-nlv6p\") pod \"maas-api-revoke-sub-c1ac44e5-xghg9\" (UID: \"6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ac44e5-xghg9" Nov 01 06:14:23.623473 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:23.623345 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1ac44e5-xghg9\" (UID: \"6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ac44e5-xghg9" Nov 01 06:14:23.624089 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:23.624068 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1ac44e5-xghg9\" (UID: \"6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ac44e5-xghg9" Nov 01 06:14:23.625753 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:23.625729 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1ac44e5-xghg9\" (UID: \"6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ac44e5-xghg9" Nov 01 06:14:23.685560 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:23.685523 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlv6p\" (UniqueName: \"kubernetes.io/projected/6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff-kube-api-access-nlv6p\") pod \"maas-api-revoke-sub-c1ac44e5-xghg9\" (UID: \"6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ac44e5-xghg9" Nov 01 06:14:23.757027 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:23.756996 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ac44e5-xghg9" Nov 01 06:14:24.138274 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:24.138248 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1ac44e5-xghg9"] Nov 01 06:14:24.139620 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:14:24.139589 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bbd5773_34f4_4fb2_b2be_ced0b62ca0ff.slice/crio-6b12dbfff9af7a10d2c1b6cae193d6b417625ada77f00e2131df248ffed1b088 WatchSource:0}: Error finding container 6b12dbfff9af7a10d2c1b6cae193d6b417625ada77f00e2131df248ffed1b088: Status 404 returned error can't find the container with id 6b12dbfff9af7a10d2c1b6cae193d6b417625ada77f00e2131df248ffed1b088 Nov 01 06:14:24.862624 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:24.862585 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ac44e5-xghg9" event={"ID":"6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff","Type":"ContainerStarted","Data":"c3ca4dd27f8aaad821fb721c3f50f220a815c4288ac91f72a94ceb219b1d74c4"} Nov 01 06:14:24.863115 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:24.862683 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ac44e5-xghg9" event={"ID":"6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff","Type":"ContainerStarted","Data":"6b12dbfff9af7a10d2c1b6cae193d6b417625ada77f00e2131df248ffed1b088"} Nov 01 06:14:24.982060 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:24.981999 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ac44e5-xghg9" podStartSLOduration=1.981985047 podStartE2EDuration="1.981985047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:14:24.979883155 +0000 UTC m=+1743.635000321" watchObservedRunningTime="2026-11-01 06:14:24.981985047 +0000 UTC m=+1743.637102214" Nov 01 06:14:25.869825 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:25.869787 5040 generic.go:356] "Generic (PLEG): container finished" podID="6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff" containerID="c3ca4dd27f8aaad821fb721c3f50f220a815c4288ac91f72a94ceb219b1d74c4" exitCode=0 Nov 01 06:14:25.870292 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:25.869831 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ac44e5-xghg9" event={"ID":"6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff","Type":"ContainerDied","Data":"c3ca4dd27f8aaad821fb721c3f50f220a815c4288ac91f72a94ceb219b1d74c4"} Nov 01 06:14:27.024402 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:27.024372 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ac44e5-xghg9" Nov 01 06:14:27.161236 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:27.160681 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff-maas-api-openshift-ca\") pod \"6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff\" (UID: \"6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff\") " Nov 01 06:14:27.161236 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:27.160885 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff-maas-api-xks-ca\") pod \"6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff\" (UID: \"6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff\") " Nov 01 06:14:27.161236 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:27.160933 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff-kube-api-access-nlv6p\" (UniqueName: \"kubernetes.io/projected/6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff-kube-api-access-nlv6p\") pod \"6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff\" (UID: \"6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff\") " Nov 01 06:14:27.161236 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:27.161046 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff-maas-api-openshift-ca" pod "6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff" (UID: "6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:14:27.161598 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:27.161332 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:27.163198 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:27.163172 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff-maas-api-xks-ca" pod "6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff" (UID: "6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:14:27.168637 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:27.168609 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff-kube-api-access-nlv6p" pod "6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff" (UID: "6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff"). InnerVolumeSpecName "kube-api-access-nlv6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:14:27.262157 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:27.262124 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:27.262157 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:27.262153 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlv6p\" (UniqueName: \"kubernetes.io/projected/6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff-kube-api-access-nlv6p\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:27.883840 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:27.883761 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ac44e5-xghg9" Nov 01 06:14:27.883840 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:27.883800 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ac44e5-xghg9" event={"ID":"6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff","Type":"ContainerDied","Data":"6b12dbfff9af7a10d2c1b6cae193d6b417625ada77f00e2131df248ffed1b088"} Nov 01 06:14:27.884020 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:27.883844 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b12dbfff9af7a10d2c1b6cae193d6b417625ada77f00e2131df248ffed1b088" Nov 01 06:14:39.974464 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:14:39.974418 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:14:40.744074 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:40.744042 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6"] Nov 01 06:14:46.906962 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:46.906874 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4dad42b-kfmjj"] Nov 01 06:14:46.907372 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:46.907352 5040 cpu_manager.go:414] "containerMap: removing container" podUID="6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff" containerName="revoke-keys" Nov 01 06:14:46.907372 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:46.907367 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff" containerName="revoke-keys" Nov 01 06:14:46.907493 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:46.907467 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff" containerName="revoke-keys" Nov 01 06:14:46.910670 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:46.910643 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4dad42b-kfmjj" Nov 01 06:14:46.968922 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:46.968805 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4dad42b-kfmjj"] Nov 01 06:14:46.977539 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:46.977492 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1ed0d77-6c9a-492c-9c07-20019eb15284-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4dad42b-kfmjj\" (UID: \"a1ed0d77-6c9a-492c-9c07-20019eb15284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4dad42b-kfmjj" Nov 01 06:14:46.977728 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:46.977588 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1ed0d77-6c9a-492c-9c07-20019eb15284-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4dad42b-kfmjj\" (UID: \"a1ed0d77-6c9a-492c-9c07-20019eb15284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4dad42b-kfmjj" Nov 01 06:14:46.977825 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:46.977728 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4cqb\" (UniqueName: \"kubernetes.io/projected/a1ed0d77-6c9a-492c-9c07-20019eb15284-kube-api-access-x4cqb\") pod \"maas-api-revoke-sub-c4dad42b-kfmjj\" (UID: \"a1ed0d77-6c9a-492c-9c07-20019eb15284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4dad42b-kfmjj" Nov 01 06:14:47.015436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:47.015396 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:14:47.015723 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:47.015533 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:14:47.015723 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:47.015556 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:14:47.015922 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:47.015576 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:14:47.079144 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:47.079109 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4cqb\" (UniqueName: \"kubernetes.io/projected/a1ed0d77-6c9a-492c-9c07-20019eb15284-kube-api-access-x4cqb\") pod \"maas-api-revoke-sub-c4dad42b-kfmjj\" (UID: \"a1ed0d77-6c9a-492c-9c07-20019eb15284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4dad42b-kfmjj" Nov 01 06:14:47.079327 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:47.079187 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1ed0d77-6c9a-492c-9c07-20019eb15284-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4dad42b-kfmjj\" (UID: \"a1ed0d77-6c9a-492c-9c07-20019eb15284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4dad42b-kfmjj" Nov 01 06:14:47.079327 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:47.079205 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1ed0d77-6c9a-492c-9c07-20019eb15284-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4dad42b-kfmjj\" (UID: \"a1ed0d77-6c9a-492c-9c07-20019eb15284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4dad42b-kfmjj" Nov 01 06:14:47.079837 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:47.079817 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1ed0d77-6c9a-492c-9c07-20019eb15284-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4dad42b-kfmjj\" (UID: \"a1ed0d77-6c9a-492c-9c07-20019eb15284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4dad42b-kfmjj" Nov 01 06:14:47.081744 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:47.081717 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1ed0d77-6c9a-492c-9c07-20019eb15284-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4dad42b-kfmjj\" (UID: \"a1ed0d77-6c9a-492c-9c07-20019eb15284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4dad42b-kfmjj" Nov 01 06:14:47.141758 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:47.141712 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4cqb\" (UniqueName: \"kubernetes.io/projected/a1ed0d77-6c9a-492c-9c07-20019eb15284-kube-api-access-x4cqb\") pod \"maas-api-revoke-sub-c4dad42b-kfmjj\" (UID: \"a1ed0d77-6c9a-492c-9c07-20019eb15284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4dad42b-kfmjj" Nov 01 06:14:47.225643 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:47.225553 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4dad42b-kfmjj" Nov 01 06:14:47.641631 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:47.626342 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4dad42b-kfmjj"] Nov 01 06:14:48.012998 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:48.012961 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4dad42b-kfmjj" event={"ID":"a1ed0d77-6c9a-492c-9c07-20019eb15284","Type":"ContainerStarted","Data":"c7dbde748ddf3a18741ec0b96e6d514f229f6818c4efdf5f1f9430e21ce73081"} Nov 01 06:14:49.020325 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:49.020283 5040 generic.go:356] "Generic (PLEG): container finished" podID="a1ed0d77-6c9a-492c-9c07-20019eb15284" containerID="fe03153a8809029929e8ff8888746579328f5fd1907317aeae7b8c4f17646fa0" exitCode=0 Nov 01 06:14:49.020937 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:49.020370 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4dad42b-kfmjj" event={"ID":"a1ed0d77-6c9a-492c-9c07-20019eb15284","Type":"ContainerDied","Data":"fe03153a8809029929e8ff8888746579328f5fd1907317aeae7b8c4f17646fa0"} Nov 01 06:14:50.380414 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:50.380384 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4dad42b-kfmjj" Nov 01 06:14:50.437554 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:50.437516 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1ed0d77-6c9a-492c-9c07-20019eb15284-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1ed0d77-6c9a-492c-9c07-20019eb15284-maas-api-xks-ca\") pod \"a1ed0d77-6c9a-492c-9c07-20019eb15284\" (UID: \"a1ed0d77-6c9a-492c-9c07-20019eb15284\") " Nov 01 06:14:50.437731 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:50.437697 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1ed0d77-6c9a-492c-9c07-20019eb15284-kube-api-access-x4cqb\" (UniqueName: \"kubernetes.io/projected/a1ed0d77-6c9a-492c-9c07-20019eb15284-kube-api-access-x4cqb\") pod \"a1ed0d77-6c9a-492c-9c07-20019eb15284\" (UID: \"a1ed0d77-6c9a-492c-9c07-20019eb15284\") " Nov 01 06:14:50.437790 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:50.437731 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1ed0d77-6c9a-492c-9c07-20019eb15284-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1ed0d77-6c9a-492c-9c07-20019eb15284-maas-api-openshift-ca\") pod \"a1ed0d77-6c9a-492c-9c07-20019eb15284\" (UID: \"a1ed0d77-6c9a-492c-9c07-20019eb15284\") " Nov 01 06:14:50.438184 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:50.438148 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1ed0d77-6c9a-492c-9c07-20019eb15284-maas-api-openshift-ca" pod "a1ed0d77-6c9a-492c-9c07-20019eb15284" (UID: "a1ed0d77-6c9a-492c-9c07-20019eb15284"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:14:50.439755 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:50.439727 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1ed0d77-6c9a-492c-9c07-20019eb15284-maas-api-xks-ca" pod "a1ed0d77-6c9a-492c-9c07-20019eb15284" (UID: "a1ed0d77-6c9a-492c-9c07-20019eb15284"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:14:50.439880 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:50.439765 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ed0d77-6c9a-492c-9c07-20019eb15284-kube-api-access-x4cqb" pod "a1ed0d77-6c9a-492c-9c07-20019eb15284" (UID: "a1ed0d77-6c9a-492c-9c07-20019eb15284"). InnerVolumeSpecName "kube-api-access-x4cqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:14:50.539383 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:50.539340 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1ed0d77-6c9a-492c-9c07-20019eb15284-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:50.539383 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:50.539376 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4cqb\" (UniqueName: \"kubernetes.io/projected/a1ed0d77-6c9a-492c-9c07-20019eb15284-kube-api-access-x4cqb\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:50.539383 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:50.539391 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1ed0d77-6c9a-492c-9c07-20019eb15284-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:51.038027 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:51.037994 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4dad42b-kfmjj" Nov 01 06:14:51.038227 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:51.037995 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4dad42b-kfmjj" event={"ID":"a1ed0d77-6c9a-492c-9c07-20019eb15284","Type":"ContainerDied","Data":"c7dbde748ddf3a18741ec0b96e6d514f229f6818c4efdf5f1f9430e21ce73081"} Nov 01 06:14:51.038227 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:51.038152 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7dbde748ddf3a18741ec0b96e6d514f229f6818c4efdf5f1f9430e21ce73081" Nov 01 06:14:55.764378 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:55.764328 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6" podUID="c2fbb701-9557-4fc1-a149-591122b79492" containerName="main" containerID="cri-o://8da601a66fe3d5783d1e8f618a7a0c4bf0127ba87cfbdee37c7e6ea0ebe45c69" gracePeriod=45 Nov 01 06:14:56.030694 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:56.030629 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6" Nov 01 06:14:56.071289 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:56.071251 5040 generic.go:356] "Generic (PLEG): container finished" podID="c2fbb701-9557-4fc1-a149-591122b79492" containerID="8da601a66fe3d5783d1e8f618a7a0c4bf0127ba87cfbdee37c7e6ea0ebe45c69" exitCode=0 Nov 01 06:14:56.071452 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:56.071294 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6" Nov 01 06:14:56.071452 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:56.071341 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6" event={"ID":"c2fbb701-9557-4fc1-a149-591122b79492","Type":"ContainerDied","Data":"8da601a66fe3d5783d1e8f618a7a0c4bf0127ba87cfbdee37c7e6ea0ebe45c69"} Nov 01 06:14:56.071452 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:56.071382 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6" event={"ID":"c2fbb701-9557-4fc1-a149-591122b79492","Type":"ContainerDied","Data":"ec45abbae19ff172000bba60982cd0f2003c756efc316f4e0983492aa8ad934e"} Nov 01 06:14:56.071452 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:56.071405 5040 scope.go:122] "RemoveContainer" containerID="8da601a66fe3d5783d1e8f618a7a0c4bf0127ba87cfbdee37c7e6ea0ebe45c69" Nov 01 06:14:56.080715 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:56.080698 5040 scope.go:122] "RemoveContainer" containerID="8da601a66fe3d5783d1e8f618a7a0c4bf0127ba87cfbdee37c7e6ea0ebe45c69" Nov 01 06:14:56.080983 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:14:56.080963 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8da601a66fe3d5783d1e8f618a7a0c4bf0127ba87cfbdee37c7e6ea0ebe45c69\": container with ID starting with 8da601a66fe3d5783d1e8f618a7a0c4bf0127ba87cfbdee37c7e6ea0ebe45c69 not found: ID does not exist" containerID="8da601a66fe3d5783d1e8f618a7a0c4bf0127ba87cfbdee37c7e6ea0ebe45c69" Nov 01 06:14:56.081069 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:56.080996 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8da601a66fe3d5783d1e8f618a7a0c4bf0127ba87cfbdee37c7e6ea0ebe45c69"} err="failed to get container status \"8da601a66fe3d5783d1e8f618a7a0c4bf0127ba87cfbdee37c7e6ea0ebe45c69\": rpc error: code = NotFound desc = could not find container \"8da601a66fe3d5783d1e8f618a7a0c4bf0127ba87cfbdee37c7e6ea0ebe45c69\": container with ID starting with 8da601a66fe3d5783d1e8f618a7a0c4bf0127ba87cfbdee37c7e6ea0ebe45c69 not found: ID does not exist" Nov 01 06:14:56.197287 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:56.197256 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2fbb701-9557-4fc1-a149-591122b79492-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2fbb701-9557-4fc1-a149-591122b79492-tmp-dir\") pod \"c2fbb701-9557-4fc1-a149-591122b79492\" (UID: \"c2fbb701-9557-4fc1-a149-591122b79492\") " Nov 01 06:14:56.197452 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:56.197307 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2fbb701-9557-4fc1-a149-591122b79492-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2fbb701-9557-4fc1-a149-591122b79492-dshm\") pod \"c2fbb701-9557-4fc1-a149-591122b79492\" (UID: \"c2fbb701-9557-4fc1-a149-591122b79492\") " Nov 01 06:14:56.197452 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:56.197358 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2fbb701-9557-4fc1-a149-591122b79492-tls-certs\" (UniqueName: \"kubernetes.io/secret/c2fbb701-9557-4fc1-a149-591122b79492-tls-certs\") pod \"c2fbb701-9557-4fc1-a149-591122b79492\" (UID: \"c2fbb701-9557-4fc1-a149-591122b79492\") " Nov 01 06:14:56.197452 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:56.197390 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2fbb701-9557-4fc1-a149-591122b79492-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2fbb701-9557-4fc1-a149-591122b79492-model-cache\") pod \"c2fbb701-9557-4fc1-a149-591122b79492\" (UID: \"c2fbb701-9557-4fc1-a149-591122b79492\") " Nov 01 06:14:56.197452 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:56.197441 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2fbb701-9557-4fc1-a149-591122b79492-home\" (UniqueName: \"kubernetes.io/empty-dir/c2fbb701-9557-4fc1-a149-591122b79492-home\") pod \"c2fbb701-9557-4fc1-a149-591122b79492\" (UID: \"c2fbb701-9557-4fc1-a149-591122b79492\") " Nov 01 06:14:56.197723 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:56.197505 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2fbb701-9557-4fc1-a149-591122b79492-kube-api-access-pvh59\" (UniqueName: \"kubernetes.io/projected/c2fbb701-9557-4fc1-a149-591122b79492-kube-api-access-pvh59\") pod \"c2fbb701-9557-4fc1-a149-591122b79492\" (UID: \"c2fbb701-9557-4fc1-a149-591122b79492\") " Nov 01 06:14:56.197723 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:56.197569 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2fbb701-9557-4fc1-a149-591122b79492-tmp-dir" pod "c2fbb701-9557-4fc1-a149-591122b79492" (UID: "c2fbb701-9557-4fc1-a149-591122b79492"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:56.197824 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:56.197734 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2fbb701-9557-4fc1-a149-591122b79492-home" pod "c2fbb701-9557-4fc1-a149-591122b79492" (UID: "c2fbb701-9557-4fc1-a149-591122b79492"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:56.197824 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:56.197751 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2fbb701-9557-4fc1-a149-591122b79492-model-cache" pod "c2fbb701-9557-4fc1-a149-591122b79492" (UID: "c2fbb701-9557-4fc1-a149-591122b79492"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:56.197900 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:56.197880 5040 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2fbb701-9557-4fc1-a149-591122b79492-model-cache\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:56.197900 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:56.197894 5040 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2fbb701-9557-4fc1-a149-591122b79492-home\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:56.197972 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:56.197905 5040 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2fbb701-9557-4fc1-a149-591122b79492-tmp-dir\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:56.199947 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:56.199922 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2fbb701-9557-4fc1-a149-591122b79492-kube-api-access-pvh59" pod "c2fbb701-9557-4fc1-a149-591122b79492" (UID: "c2fbb701-9557-4fc1-a149-591122b79492"). InnerVolumeSpecName "kube-api-access-pvh59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:14:56.199947 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:56.199934 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2fbb701-9557-4fc1-a149-591122b79492-tls-certs" pod "c2fbb701-9557-4fc1-a149-591122b79492" (UID: "c2fbb701-9557-4fc1-a149-591122b79492"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:14:56.200126 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:56.199996 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2fbb701-9557-4fc1-a149-591122b79492-dshm" pod "c2fbb701-9557-4fc1-a149-591122b79492" (UID: "c2fbb701-9557-4fc1-a149-591122b79492"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:56.299161 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:56.299078 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvh59\" (UniqueName: \"kubernetes.io/projected/c2fbb701-9557-4fc1-a149-591122b79492-kube-api-access-pvh59\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:56.299161 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:56.299108 5040 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2fbb701-9557-4fc1-a149-591122b79492-dshm\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:56.299161 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:56.299117 5040 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2fbb701-9557-4fc1-a149-591122b79492-tls-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:56.494322 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:56.494285 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6"] Nov 01 06:14:56.500677 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:56.500650 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-3578ce/praxis-route-3578ce-kserve-57f8955b99-jhqf6"] Nov 01 06:14:57.515453 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:14:57.515418 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2fbb701-9557-4fc1-a149-591122b79492" path="/var/lib/kubelet/pods/c2fbb701-9557-4fc1-a149-591122b79492/volumes" Nov 01 06:15:00.176116 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.175941 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891895-h46r4"] Nov 01 06:15:00.176611 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.176459 5040 cpu_manager.go:414] "containerMap: removing container" podUID="a1ed0d77-6c9a-492c-9c07-20019eb15284" containerName="revoke-keys" Nov 01 06:15:00.176611 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.176473 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1ed0d77-6c9a-492c-9c07-20019eb15284" containerName="revoke-keys" Nov 01 06:15:00.176611 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.176530 5040 cpu_manager.go:414] "containerMap: removing container" podUID="c2fbb701-9557-4fc1-a149-591122b79492" containerName="main" Nov 01 06:15:00.176611 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.176539 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2fbb701-9557-4fc1-a149-591122b79492" containerName="main" Nov 01 06:15:00.176797 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.176652 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2fbb701-9557-4fc1-a149-591122b79492" containerName="main" Nov 01 06:15:00.176797 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.176667 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1ed0d77-6c9a-492c-9c07-20019eb15284" containerName="revoke-keys" Nov 01 06:15:00.181222 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.181204 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec5596-29891895-nz6tn"] Nov 01 06:15:00.181347 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.181327 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891895-h46r4" Nov 01 06:15:00.184785 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.184763 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891895-w8t88"] Nov 01 06:15:00.184904 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.184873 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec5596-29891895-nz6tn" Nov 01 06:15:00.188199 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.188180 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-w8t88" Nov 01 06:15:00.226393 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.226362 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891895-bb2zn"] Nov 01 06:15:00.229810 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.229793 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3578ce-29891895-5vnj9"] Nov 01 06:15:00.229925 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.229908 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891895-bb2zn" Nov 01 06:15:00.233634 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.233611 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891895-p4bzz"] Nov 01 06:15:00.233761 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.233712 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3578ce-29891895-5vnj9" Nov 01 06:15:00.234528 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.234506 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:15:00.237494 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.237456 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891895-mvbfb"] Nov 01 06:15:00.237599 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.237560 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891895-p4bzz" Nov 01 06:15:00.240923 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.240906 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891895-mvbfb" Nov 01 06:15:00.274503 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.274449 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:15:00.274675 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.274576 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:15:00.278412 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.278385 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891895-h46r4"] Nov 01 06:15:00.281181 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.281158 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec5596-29891895-nz6tn"] Nov 01 06:15:00.285194 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.285167 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891895-w8t88"] Nov 01 06:15:00.346501 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.341121 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlcpt\" (UniqueName: \"kubernetes.io/projected/78e5172d-422f-4082-9f96-15f1ab8b49eb-kube-api-access-jlcpt\") pod \"maas-api-key-cleanup-29891895-w8t88\" (UID: \"78e5172d-422f-4082-9f96-15f1ab8b49eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-w8t88" Nov 01 06:15:00.346501 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.341177 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1f2aca-3d31-4c8d-b83a-ca20a360e75e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-3578ce-29891895-5vnj9\" (UID: \"3b1f2aca-3d31-4c8d-b83a-ca20a360e75e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3578ce-29891895-5vnj9" Nov 01 06:15:00.346501 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.341279 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjvr8\" (UniqueName: \"kubernetes.io/projected/ec119deb-0b81-447c-af06-9464177b7a01-kube-api-access-sjvr8\") pod \"maas-api-key-cleanup-e2e-praxis-b-ec5596-29891895-nz6tn\" (UID: \"ec119deb-0b81-447c-af06-9464177b7a01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec5596-29891895-nz6tn" Nov 01 06:15:00.346501 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.341309 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb856\" (UniqueName: \"kubernetes.io/projected/7b7f1411-aaf9-4d3b-91c0-40f875416c57-kube-api-access-kb856\") pod \"maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891895-bb2zn\" (UID: \"7b7f1411-aaf9-4d3b-91c0-40f875416c57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891895-bb2zn" Nov 01 06:15:00.346501 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.341354 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25021d94-5042-4ee0-b69f-12ceb0564036-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-3986bf-29891895-mvbfb\" (UID: \"25021d94-5042-4ee0-b69f-12ceb0564036\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891895-mvbfb" Nov 01 06:15:00.346501 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.341395 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f7p8\" (UniqueName: \"kubernetes.io/projected/3b1f2aca-3d31-4c8d-b83a-ca20a360e75e-kube-api-access-4f7p8\") pod \"maas-api-key-cleanup-e2e-praxis-a-3578ce-29891895-5vnj9\" (UID: \"3b1f2aca-3d31-4c8d-b83a-ca20a360e75e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3578ce-29891895-5vnj9" Nov 01 06:15:00.346501 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.341417 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9bjq\" (UniqueName: \"kubernetes.io/projected/25021d94-5042-4ee0-b69f-12ceb0564036-kube-api-access-d9bjq\") pod \"maas-api-key-cleanup-e2e-extmod-b-3986bf-29891895-mvbfb\" (UID: \"25021d94-5042-4ee0-b69f-12ceb0564036\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891895-mvbfb" Nov 01 06:15:00.346501 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.341453 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6610260-baea-4921-9a3a-756fbf18eb3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891895-h46r4\" (UID: \"c6610260-baea-4921-9a3a-756fbf18eb3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891895-h46r4" Nov 01 06:15:00.346501 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.341576 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec119deb-0b81-447c-af06-9464177b7a01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-ec5596-29891895-nz6tn\" (UID: \"ec119deb-0b81-447c-af06-9464177b7a01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec5596-29891895-nz6tn" Nov 01 06:15:00.346501 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.341633 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwstr\" (UniqueName: \"kubernetes.io/projected/0d62a50d-6bb5-4200-bc69-70deaa07d3d3-kube-api-access-pwstr\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d25db-29891895-p4bzz\" (UID: \"0d62a50d-6bb5-4200-bc69-70deaa07d3d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891895-p4bzz" Nov 01 06:15:00.346501 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.341671 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78e5172d-422f-4082-9f96-15f1ab8b49eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891895-w8t88\" (UID: \"78e5172d-422f-4082-9f96-15f1ab8b49eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-w8t88" Nov 01 06:15:00.346501 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.341703 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d62a50d-6bb5-4200-bc69-70deaa07d3d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d25db-29891895-p4bzz\" (UID: \"0d62a50d-6bb5-4200-bc69-70deaa07d3d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891895-p4bzz" Nov 01 06:15:00.346501 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.341755 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b7f1411-aaf9-4d3b-91c0-40f875416c57-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891895-bb2zn\" (UID: \"7b7f1411-aaf9-4d3b-91c0-40f875416c57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891895-bb2zn" Nov 01 06:15:00.346501 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.341782 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8dv2\" (UniqueName: \"kubernetes.io/projected/c6610260-baea-4921-9a3a-756fbf18eb3e-kube-api-access-m8dv2\") pod \"maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891895-h46r4\" (UID: \"c6610260-baea-4921-9a3a-756fbf18eb3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891895-h46r4" Nov 01 06:15:00.347604 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.347574 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891895-p4bzz"] Nov 01 06:15:00.353080 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.351069 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891895-bb2zn"] Nov 01 06:15:00.355753 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.355727 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891895-mvbfb"] Nov 01 06:15:00.360525 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.358012 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3578ce-29891895-5vnj9"] Nov 01 06:15:00.442924 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.442833 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6610260-baea-4921-9a3a-756fbf18eb3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891895-h46r4\" (UID: \"c6610260-baea-4921-9a3a-756fbf18eb3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891895-h46r4" Nov 01 06:15:00.442924 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.442877 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec119deb-0b81-447c-af06-9464177b7a01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-ec5596-29891895-nz6tn\" (UID: \"ec119deb-0b81-447c-af06-9464177b7a01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec5596-29891895-nz6tn" Nov 01 06:15:00.442924 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.442898 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwstr\" (UniqueName: \"kubernetes.io/projected/0d62a50d-6bb5-4200-bc69-70deaa07d3d3-kube-api-access-pwstr\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d25db-29891895-p4bzz\" (UID: \"0d62a50d-6bb5-4200-bc69-70deaa07d3d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891895-p4bzz" Nov 01 06:15:00.443228 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.443017 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78e5172d-422f-4082-9f96-15f1ab8b49eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891895-w8t88\" (UID: \"78e5172d-422f-4082-9f96-15f1ab8b49eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-w8t88" Nov 01 06:15:00.443228 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.443058 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d62a50d-6bb5-4200-bc69-70deaa07d3d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d25db-29891895-p4bzz\" (UID: \"0d62a50d-6bb5-4200-bc69-70deaa07d3d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891895-p4bzz" Nov 01 06:15:00.443228 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.443111 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b7f1411-aaf9-4d3b-91c0-40f875416c57-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891895-bb2zn\" (UID: \"7b7f1411-aaf9-4d3b-91c0-40f875416c57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891895-bb2zn" Nov 01 06:15:00.443228 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.443137 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8dv2\" (UniqueName: \"kubernetes.io/projected/c6610260-baea-4921-9a3a-756fbf18eb3e-kube-api-access-m8dv2\") pod \"maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891895-h46r4\" (UID: \"c6610260-baea-4921-9a3a-756fbf18eb3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891895-h46r4" Nov 01 06:15:00.443228 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.443218 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlcpt\" (UniqueName: \"kubernetes.io/projected/78e5172d-422f-4082-9f96-15f1ab8b49eb-kube-api-access-jlcpt\") pod \"maas-api-key-cleanup-29891895-w8t88\" (UID: \"78e5172d-422f-4082-9f96-15f1ab8b49eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-w8t88" Nov 01 06:15:00.443471 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.443380 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1f2aca-3d31-4c8d-b83a-ca20a360e75e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-3578ce-29891895-5vnj9\" (UID: \"3b1f2aca-3d31-4c8d-b83a-ca20a360e75e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3578ce-29891895-5vnj9" Nov 01 06:15:00.443471 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.443444 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjvr8\" (UniqueName: \"kubernetes.io/projected/ec119deb-0b81-447c-af06-9464177b7a01-kube-api-access-sjvr8\") pod \"maas-api-key-cleanup-e2e-praxis-b-ec5596-29891895-nz6tn\" (UID: \"ec119deb-0b81-447c-af06-9464177b7a01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec5596-29891895-nz6tn" Nov 01 06:15:00.443613 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.443499 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb856\" (UniqueName: \"kubernetes.io/projected/7b7f1411-aaf9-4d3b-91c0-40f875416c57-kube-api-access-kb856\") pod \"maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891895-bb2zn\" (UID: \"7b7f1411-aaf9-4d3b-91c0-40f875416c57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891895-bb2zn" Nov 01 06:15:00.443613 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.443550 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25021d94-5042-4ee0-b69f-12ceb0564036-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-3986bf-29891895-mvbfb\" (UID: \"25021d94-5042-4ee0-b69f-12ceb0564036\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891895-mvbfb" Nov 01 06:15:00.443613 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.443600 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f7p8\" (UniqueName: \"kubernetes.io/projected/3b1f2aca-3d31-4c8d-b83a-ca20a360e75e-kube-api-access-4f7p8\") pod \"maas-api-key-cleanup-e2e-praxis-a-3578ce-29891895-5vnj9\" (UID: \"3b1f2aca-3d31-4c8d-b83a-ca20a360e75e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3578ce-29891895-5vnj9" Nov 01 06:15:00.443783 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.443624 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9bjq\" (UniqueName: \"kubernetes.io/projected/25021d94-5042-4ee0-b69f-12ceb0564036-kube-api-access-d9bjq\") pod \"maas-api-key-cleanup-e2e-extmod-b-3986bf-29891895-mvbfb\" (UID: \"25021d94-5042-4ee0-b69f-12ceb0564036\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891895-mvbfb" Nov 01 06:15:00.443874 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.443849 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec119deb-0b81-447c-af06-9464177b7a01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-ec5596-29891895-nz6tn\" (UID: \"ec119deb-0b81-447c-af06-9464177b7a01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec5596-29891895-nz6tn" Nov 01 06:15:00.443938 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.443861 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6610260-baea-4921-9a3a-756fbf18eb3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891895-h46r4\" (UID: \"c6610260-baea-4921-9a3a-756fbf18eb3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891895-h46r4" Nov 01 06:15:00.443938 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.443850 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78e5172d-422f-4082-9f96-15f1ab8b49eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891895-w8t88\" (UID: \"78e5172d-422f-4082-9f96-15f1ab8b49eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-w8t88" Nov 01 06:15:00.443938 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.443904 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d62a50d-6bb5-4200-bc69-70deaa07d3d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d25db-29891895-p4bzz\" (UID: \"0d62a50d-6bb5-4200-bc69-70deaa07d3d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891895-p4bzz" Nov 01 06:15:00.443938 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.443925 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b7f1411-aaf9-4d3b-91c0-40f875416c57-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891895-bb2zn\" (UID: \"7b7f1411-aaf9-4d3b-91c0-40f875416c57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891895-bb2zn" Nov 01 06:15:00.444269 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.444249 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1f2aca-3d31-4c8d-b83a-ca20a360e75e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-3578ce-29891895-5vnj9\" (UID: \"3b1f2aca-3d31-4c8d-b83a-ca20a360e75e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3578ce-29891895-5vnj9" Nov 01 06:15:00.444507 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.444472 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25021d94-5042-4ee0-b69f-12ceb0564036-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-3986bf-29891895-mvbfb\" (UID: \"25021d94-5042-4ee0-b69f-12ceb0564036\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891895-mvbfb" Nov 01 06:15:00.505516 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.505457 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9bjq\" (UniqueName: \"kubernetes.io/projected/25021d94-5042-4ee0-b69f-12ceb0564036-kube-api-access-d9bjq\") pod \"maas-api-key-cleanup-e2e-extmod-b-3986bf-29891895-mvbfb\" (UID: \"25021d94-5042-4ee0-b69f-12ceb0564036\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891895-mvbfb" Nov 01 06:15:00.505703 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.505459 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlcpt\" (UniqueName: \"kubernetes.io/projected/78e5172d-422f-4082-9f96-15f1ab8b49eb-kube-api-access-jlcpt\") pod \"maas-api-key-cleanup-29891895-w8t88\" (UID: \"78e5172d-422f-4082-9f96-15f1ab8b49eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-w8t88" Nov 01 06:15:00.505901 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.505877 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8dv2\" (UniqueName: \"kubernetes.io/projected/c6610260-baea-4921-9a3a-756fbf18eb3e-kube-api-access-m8dv2\") pod \"maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891895-h46r4\" (UID: \"c6610260-baea-4921-9a3a-756fbf18eb3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891895-h46r4" Nov 01 06:15:00.505982 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.505886 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb856\" (UniqueName: \"kubernetes.io/projected/7b7f1411-aaf9-4d3b-91c0-40f875416c57-kube-api-access-kb856\") pod \"maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891895-bb2zn\" (UID: \"7b7f1411-aaf9-4d3b-91c0-40f875416c57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891895-bb2zn" Nov 01 06:15:00.506040 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.506020 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwstr\" (UniqueName: \"kubernetes.io/projected/0d62a50d-6bb5-4200-bc69-70deaa07d3d3-kube-api-access-pwstr\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d25db-29891895-p4bzz\" (UID: \"0d62a50d-6bb5-4200-bc69-70deaa07d3d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891895-p4bzz" Nov 01 06:15:00.506584 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.506551 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f7p8\" (UniqueName: \"kubernetes.io/projected/3b1f2aca-3d31-4c8d-b83a-ca20a360e75e-kube-api-access-4f7p8\") pod \"maas-api-key-cleanup-e2e-praxis-a-3578ce-29891895-5vnj9\" (UID: \"3b1f2aca-3d31-4c8d-b83a-ca20a360e75e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3578ce-29891895-5vnj9" Nov 01 06:15:00.506688 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.506636 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjvr8\" (UniqueName: \"kubernetes.io/projected/ec119deb-0b81-447c-af06-9464177b7a01-kube-api-access-sjvr8\") pod \"maas-api-key-cleanup-e2e-praxis-b-ec5596-29891895-nz6tn\" (UID: \"ec119deb-0b81-447c-af06-9464177b7a01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec5596-29891895-nz6tn" Nov 01 06:15:00.508417 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.508395 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-w8t88" Nov 01 06:15:00.546676 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.545527 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891895-bb2zn" Nov 01 06:15:00.556341 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.553953 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3578ce-29891895-5vnj9" Nov 01 06:15:00.571519 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.571162 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891895-p4bzz" Nov 01 06:15:00.577700 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.577518 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891895-mvbfb" Nov 01 06:15:00.798057 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.798023 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891895-h46r4" Nov 01 06:15:00.804103 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.804079 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec5596-29891895-nz6tn" Nov 01 06:15:00.914876 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:15:00.914840 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e5172d_422f_4082_9f96_15f1ab8b49eb.slice/crio-f393f823c64a5804cdb28ae66c0a0b6eb007440499076463f8c8db8b76a3dc6c WatchSource:0}: Error finding container f393f823c64a5804cdb28ae66c0a0b6eb007440499076463f8c8db8b76a3dc6c: Status 404 returned error can't find the container with id f393f823c64a5804cdb28ae66c0a0b6eb007440499076463f8c8db8b76a3dc6c Nov 01 06:15:00.919695 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.919668 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891895-w8t88"] Nov 01 06:15:00.968146 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:00.968109 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891895-bb2zn"] Nov 01 06:15:00.969807 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:15:00.969783 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b7f1411_aaf9_4d3b_91c0_40f875416c57.slice/crio-1b2874a77a1123cde474f9652b628c36bd7efa87a9705266e2c33a712c15f9d5 WatchSource:0}: Error finding container 1b2874a77a1123cde474f9652b628c36bd7efa87a9705266e2c33a712c15f9d5: Status 404 returned error can't find the container with id 1b2874a77a1123cde474f9652b628c36bd7efa87a9705266e2c33a712c15f9d5 Nov 01 06:15:01.016767 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:01.016741 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3578ce-29891895-5vnj9"] Nov 01 06:15:01.018303 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:15:01.018273 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b1f2aca_3d31_4c8d_b83a_ca20a360e75e.slice/crio-8009e9cb0a652f927d8c05915234544a629820475fdb0b2d5ca9fb19203193bb WatchSource:0}: Error finding container 8009e9cb0a652f927d8c05915234544a629820475fdb0b2d5ca9fb19203193bb: Status 404 returned error can't find the container with id 8009e9cb0a652f927d8c05915234544a629820475fdb0b2d5ca9fb19203193bb Nov 01 06:15:01.024966 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:15:01.024944 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25021d94_5042_4ee0_b69f_12ceb0564036.slice/crio-b47f50ebf503f9e991fcb567e7eef8cbada270a5041775b0e655e4c9b06082d5 WatchSource:0}: Error finding container b47f50ebf503f9e991fcb567e7eef8cbada270a5041775b0e655e4c9b06082d5: Status 404 returned error can't find the container with id b47f50ebf503f9e991fcb567e7eef8cbada270a5041775b0e655e4c9b06082d5 Nov 01 06:15:01.025332 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:01.025312 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891895-mvbfb"] Nov 01 06:15:01.104010 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:01.103968 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891895-mvbfb" event={"ID":"25021d94-5042-4ee0-b69f-12ceb0564036","Type":"ContainerStarted","Data":"b47f50ebf503f9e991fcb567e7eef8cbada270a5041775b0e655e4c9b06082d5"} Nov 01 06:15:01.105383 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:01.105353 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891895-bb2zn" event={"ID":"7b7f1411-aaf9-4d3b-91c0-40f875416c57","Type":"ContainerStarted","Data":"1b2874a77a1123cde474f9652b628c36bd7efa87a9705266e2c33a712c15f9d5"} Nov 01 06:15:01.106611 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:01.106590 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-w8t88" event={"ID":"78e5172d-422f-4082-9f96-15f1ab8b49eb","Type":"ContainerStarted","Data":"f393f823c64a5804cdb28ae66c0a0b6eb007440499076463f8c8db8b76a3dc6c"} Nov 01 06:15:01.107732 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:01.107711 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3578ce-29891895-5vnj9" event={"ID":"3b1f2aca-3d31-4c8d-b83a-ca20a360e75e","Type":"ContainerStarted","Data":"8009e9cb0a652f927d8c05915234544a629820475fdb0b2d5ca9fb19203193bb"} Nov 01 06:15:01.144414 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:01.144387 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891895-p4bzz"] Nov 01 06:15:01.180591 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:01.180563 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891895-h46r4"] Nov 01 06:15:01.187069 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:01.187045 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec5596-29891895-nz6tn"] Nov 01 06:15:01.197152 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:15:01.197111 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d62a50d_6bb5_4200_bc69_70deaa07d3d3.slice/crio-2947d2e6837f3d692d84db1de6d3ae802c4859384b782fb207b1c00219b0a7d4 WatchSource:0}: Error finding container 2947d2e6837f3d692d84db1de6d3ae802c4859384b782fb207b1c00219b0a7d4: Status 404 returned error can't find the container with id 2947d2e6837f3d692d84db1de6d3ae802c4859384b782fb207b1c00219b0a7d4 Nov 01 06:15:02.118533 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:02.118501 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891895-p4bzz" event={"ID":"0d62a50d-6bb5-4200-bc69-70deaa07d3d3","Type":"ContainerStarted","Data":"5fb7adddac372c874f485b11f046c492bb7dae4dd2e94f96a5291242f2ef5ec2"} Nov 01 06:15:02.118659 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:02.118569 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891895-p4bzz" event={"ID":"0d62a50d-6bb5-4200-bc69-70deaa07d3d3","Type":"ContainerStarted","Data":"2947d2e6837f3d692d84db1de6d3ae802c4859384b782fb207b1c00219b0a7d4"} Nov 01 06:15:02.120146 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:02.120119 5040 generic.go:356] "Generic (PLEG): container finished" podID="78e5172d-422f-4082-9f96-15f1ab8b49eb" containerID="20908789c7e4118e1af5ecac521202e785d595bf3d68f2fbfbc9eb23cdaa8f5d" exitCode=60 Nov 01 06:15:02.120230 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:02.120191 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-w8t88" event={"ID":"78e5172d-422f-4082-9f96-15f1ab8b49eb","Type":"ContainerDied","Data":"20908789c7e4118e1af5ecac521202e785d595bf3d68f2fbfbc9eb23cdaa8f5d"} Nov 01 06:15:02.120470 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:02.120450 5040 scope.go:122] "RemoveContainer" containerID="20908789c7e4118e1af5ecac521202e785d595bf3d68f2fbfbc9eb23cdaa8f5d" Nov 01 06:15:02.121814 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:02.121797 5040 generic.go:356] "Generic (PLEG): container finished" podID="ec119deb-0b81-447c-af06-9464177b7a01" containerID="82e29c1e5e40b607cec3101b50e1462a2ffb4c22e0c1664261ca2892403ce468" exitCode=60 Nov 01 06:15:02.121899 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:02.121877 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec5596-29891895-nz6tn" event={"ID":"ec119deb-0b81-447c-af06-9464177b7a01","Type":"ContainerDied","Data":"82e29c1e5e40b607cec3101b50e1462a2ffb4c22e0c1664261ca2892403ce468"} Nov 01 06:15:02.121969 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:02.121926 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec5596-29891895-nz6tn" event={"ID":"ec119deb-0b81-447c-af06-9464177b7a01","Type":"ContainerStarted","Data":"21e4a8a8097d63fbad5b834d6092f460b7fb059c4d8d3dc3b392f5dfa361c739"} Nov 01 06:15:02.122081 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:02.122064 5040 scope.go:122] "RemoveContainer" containerID="82e29c1e5e40b607cec3101b50e1462a2ffb4c22e0c1664261ca2892403ce468" Nov 01 06:15:02.123577 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:02.123557 5040 generic.go:356] "Generic (PLEG): container finished" podID="3b1f2aca-3d31-4c8d-b83a-ca20a360e75e" containerID="09c04c6ff67b28d420634c18c384d5bf58d108e2170231fe2fdbd48e0fc32660" exitCode=60 Nov 01 06:15:02.123688 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:02.123661 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3578ce-29891895-5vnj9" event={"ID":"3b1f2aca-3d31-4c8d-b83a-ca20a360e75e","Type":"ContainerDied","Data":"09c04c6ff67b28d420634c18c384d5bf58d108e2170231fe2fdbd48e0fc32660"} Nov 01 06:15:02.123839 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:02.123819 5040 scope.go:122] "RemoveContainer" containerID="09c04c6ff67b28d420634c18c384d5bf58d108e2170231fe2fdbd48e0fc32660" Nov 01 06:15:02.128804 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:02.125686 5040 generic.go:356] "Generic (PLEG): container finished" podID="25021d94-5042-4ee0-b69f-12ceb0564036" containerID="9e6799bd034721d070686b3ecb8f5393f8deb9bc84227fdc612acdb3abd4480a" exitCode=60 Nov 01 06:15:02.128804 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:02.126235 5040 scope.go:122] "RemoveContainer" containerID="9e6799bd034721d070686b3ecb8f5393f8deb9bc84227fdc612acdb3abd4480a" Nov 01 06:15:02.128804 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:02.126386 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891895-mvbfb" event={"ID":"25021d94-5042-4ee0-b69f-12ceb0564036","Type":"ContainerDied","Data":"9e6799bd034721d070686b3ecb8f5393f8deb9bc84227fdc612acdb3abd4480a"} Nov 01 06:15:02.136840 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:02.136816 5040 generic.go:356] "Generic (PLEG): container finished" podID="7b7f1411-aaf9-4d3b-91c0-40f875416c57" containerID="5bf7ced2675d47d23a2989acad7590ae9c48c8ea67533c58b5806f9d707bed9a" exitCode=60 Nov 01 06:15:02.136956 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:02.136901 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891895-bb2zn" event={"ID":"7b7f1411-aaf9-4d3b-91c0-40f875416c57","Type":"ContainerDied","Data":"5bf7ced2675d47d23a2989acad7590ae9c48c8ea67533c58b5806f9d707bed9a"} Nov 01 06:15:02.137132 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:02.137119 5040 scope.go:122] "RemoveContainer" containerID="5bf7ced2675d47d23a2989acad7590ae9c48c8ea67533c58b5806f9d707bed9a" Nov 01 06:15:02.144611 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:02.144563 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891895-h46r4" event={"ID":"c6610260-baea-4921-9a3a-756fbf18eb3e","Type":"ContainerStarted","Data":"0e608e9aed09d1c3716392695c6e2a7fd48c5243376e707facafe28b97b37097"} Nov 01 06:15:02.144611 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:02.144609 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891895-h46r4" event={"ID":"c6610260-baea-4921-9a3a-756fbf18eb3e","Type":"ContainerStarted","Data":"e91db8397a0f833cdcc4d67e220502b2b4981a8c1a00db524e6c20de8d00fc8e"} Nov 01 06:15:02.237355 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:02.237296 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891895-p4bzz" podStartSLOduration=2.237281604 podStartE2EDuration="2.237281604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:15:02.234629557 +0000 UTC m=+1780.889746726" watchObservedRunningTime="2026-11-01 06:15:02.237281604 +0000 UTC m=+1780.892398772" Nov 01 06:15:02.818626 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:02.818578 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891895-h46r4" podStartSLOduration=2.818564554 podStartE2EDuration="2.818564554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:15:02.816520381 +0000 UTC m=+1781.471637549" watchObservedRunningTime="2026-11-01 06:15:02.818564554 +0000 UTC m=+1781.473681766" Nov 01 06:15:03.152996 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:03.152956 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3578ce-29891895-5vnj9" event={"ID":"3b1f2aca-3d31-4c8d-b83a-ca20a360e75e","Type":"ContainerStarted","Data":"246eaeff230ea6909723d9e9159003431958942f6fd0eb8df1e7afd756dfb7cc"} Nov 01 06:15:03.154957 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:03.154928 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891895-mvbfb" event={"ID":"25021d94-5042-4ee0-b69f-12ceb0564036","Type":"ContainerStarted","Data":"8be3b9ae783244e7466209862e7af9903e4920b5b2220c9a79b53e10ea3f3a68"} Nov 01 06:15:03.156702 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:03.156679 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891895-bb2zn" event={"ID":"7b7f1411-aaf9-4d3b-91c0-40f875416c57","Type":"ContainerStarted","Data":"7094cd180e5c51cad724f59d57a02dbaf51b5becc8b7686303e6d740aa27b26b"} Nov 01 06:15:03.158188 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:03.158166 5040 generic.go:356] "Generic (PLEG): container finished" podID="c6610260-baea-4921-9a3a-756fbf18eb3e" containerID="0e608e9aed09d1c3716392695c6e2a7fd48c5243376e707facafe28b97b37097" exitCode=60 Nov 01 06:15:03.158307 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:03.158233 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891895-h46r4" event={"ID":"c6610260-baea-4921-9a3a-756fbf18eb3e","Type":"ContainerDied","Data":"0e608e9aed09d1c3716392695c6e2a7fd48c5243376e707facafe28b97b37097"} Nov 01 06:15:03.158427 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:03.158410 5040 scope.go:122] "RemoveContainer" containerID="0e608e9aed09d1c3716392695c6e2a7fd48c5243376e707facafe28b97b37097" Nov 01 06:15:03.159930 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:03.159868 5040 generic.go:356] "Generic (PLEG): container finished" podID="0d62a50d-6bb5-4200-bc69-70deaa07d3d3" containerID="5fb7adddac372c874f485b11f046c492bb7dae4dd2e94f96a5291242f2ef5ec2" exitCode=60 Nov 01 06:15:03.160000 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:03.159934 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891895-p4bzz" event={"ID":"0d62a50d-6bb5-4200-bc69-70deaa07d3d3","Type":"ContainerDied","Data":"5fb7adddac372c874f485b11f046c492bb7dae4dd2e94f96a5291242f2ef5ec2"} Nov 01 06:15:03.160154 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:03.160133 5040 scope.go:122] "RemoveContainer" containerID="5fb7adddac372c874f485b11f046c492bb7dae4dd2e94f96a5291242f2ef5ec2" Nov 01 06:15:03.162636 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:03.162614 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-w8t88" event={"ID":"78e5172d-422f-4082-9f96-15f1ab8b49eb","Type":"ContainerStarted","Data":"7eef78eb88c0ef378cfbd503bec531e71cb4d6badebd133490896d2f17d3ca6b"} Nov 01 06:15:03.164506 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:03.164459 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec5596-29891895-nz6tn" event={"ID":"ec119deb-0b81-447c-af06-9464177b7a01","Type":"ContainerStarted","Data":"7da4fe4587acd3e8f11974adfff2574213aa3be18cb6648b367c267ffda757d0"} Nov 01 06:15:03.270079 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:03.270033 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3578ce-29891895-5vnj9" podStartSLOduration=3.270019032 podStartE2EDuration="3.270019032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:15:03.269306619 +0000 UTC m=+1781.924423786" watchObservedRunningTime="2026-11-01 06:15:03.270019032 +0000 UTC m=+1781.925136199" Nov 01 06:15:03.389986 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:03.389923 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-w8t88" podStartSLOduration=3.389905517 podStartE2EDuration="3.389905517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:15:03.385409012 +0000 UTC m=+1782.040526178" watchObservedRunningTime="2026-11-01 06:15:03.389905517 +0000 UTC m=+1782.045022687" Nov 01 06:15:03.507949 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:03.507882 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891895-mvbfb" podStartSLOduration=3.507862497 podStartE2EDuration="3.507862497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:15:03.504819249 +0000 UTC m=+1782.159936419" watchObservedRunningTime="2026-11-01 06:15:03.507862497 +0000 UTC m=+1782.162979664" Nov 01 06:15:03.739322 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:03.739267 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891895-bb2zn" podStartSLOduration=3.739250787 podStartE2EDuration="3.739250787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:15:03.736739534 +0000 UTC m=+1782.391856702" watchObservedRunningTime="2026-11-01 06:15:03.739250787 +0000 UTC m=+1782.394367954" Nov 01 06:15:03.973198 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:03.973129 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec5596-29891895-nz6tn" podStartSLOduration=3.973109949 podStartE2EDuration="3.973109949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:15:03.97184855 +0000 UTC m=+1782.626965717" watchObservedRunningTime="2026-11-01 06:15:03.973109949 +0000 UTC m=+1782.628227115" Nov 01 06:15:04.173912 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:04.173880 5040 generic.go:356] "Generic (PLEG): container finished" podID="3b1f2aca-3d31-4c8d-b83a-ca20a360e75e" containerID="246eaeff230ea6909723d9e9159003431958942f6fd0eb8df1e7afd756dfb7cc" exitCode=60 Nov 01 06:15:04.174087 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:04.173957 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3578ce-29891895-5vnj9" event={"ID":"3b1f2aca-3d31-4c8d-b83a-ca20a360e75e","Type":"ContainerDied","Data":"246eaeff230ea6909723d9e9159003431958942f6fd0eb8df1e7afd756dfb7cc"} Nov 01 06:15:04.174087 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:04.174013 5040 scope.go:122] "RemoveContainer" containerID="09c04c6ff67b28d420634c18c384d5bf58d108e2170231fe2fdbd48e0fc32660" Nov 01 06:15:04.174229 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:04.174214 5040 scope.go:122] "RemoveContainer" containerID="246eaeff230ea6909723d9e9159003431958942f6fd0eb8df1e7afd756dfb7cc" Nov 01 06:15:04.174528 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:15:04.174475 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-3578ce-29891895-5vnj9_odh-ai-gateway-infra(3b1f2aca-3d31-4c8d-b83a-ca20a360e75e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3578ce-29891895-5vnj9" podUID="3b1f2aca-3d31-4c8d-b83a-ca20a360e75e" Nov 01 06:15:04.176151 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:04.176132 5040 generic.go:356] "Generic (PLEG): container finished" podID="25021d94-5042-4ee0-b69f-12ceb0564036" containerID="8be3b9ae783244e7466209862e7af9903e4920b5b2220c9a79b53e10ea3f3a68" exitCode=60 Nov 01 06:15:04.176266 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:04.176194 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891895-mvbfb" event={"ID":"25021d94-5042-4ee0-b69f-12ceb0564036","Type":"ContainerDied","Data":"8be3b9ae783244e7466209862e7af9903e4920b5b2220c9a79b53e10ea3f3a68"} Nov 01 06:15:04.176456 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:04.176440 5040 scope.go:122] "RemoveContainer" containerID="8be3b9ae783244e7466209862e7af9903e4920b5b2220c9a79b53e10ea3f3a68" Nov 01 06:15:04.176755 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:15:04.176737 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-3986bf-29891895-mvbfb_odh-ai-gateway-infra(25021d94-5042-4ee0-b69f-12ceb0564036)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891895-mvbfb" podUID="25021d94-5042-4ee0-b69f-12ceb0564036" Nov 01 06:15:04.178098 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:04.178077 5040 generic.go:356] "Generic (PLEG): container finished" podID="7b7f1411-aaf9-4d3b-91c0-40f875416c57" containerID="7094cd180e5c51cad724f59d57a02dbaf51b5becc8b7686303e6d740aa27b26b" exitCode=60 Nov 01 06:15:04.178176 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:04.178143 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891895-bb2zn" event={"ID":"7b7f1411-aaf9-4d3b-91c0-40f875416c57","Type":"ContainerDied","Data":"7094cd180e5c51cad724f59d57a02dbaf51b5becc8b7686303e6d740aa27b26b"} Nov 01 06:15:04.178408 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:04.178388 5040 scope.go:122] "RemoveContainer" containerID="7094cd180e5c51cad724f59d57a02dbaf51b5becc8b7686303e6d740aa27b26b" Nov 01 06:15:04.178647 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:15:04.178631 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891895-bb2zn_odh-ai-gateway-infra(7b7f1411-aaf9-4d3b-91c0-40f875416c57)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891895-bb2zn" podUID="7b7f1411-aaf9-4d3b-91c0-40f875416c57" Nov 01 06:15:04.179992 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:04.179974 5040 generic.go:356] "Generic (PLEG): container finished" podID="c6610260-baea-4921-9a3a-756fbf18eb3e" containerID="97bbb6b10a65496d10cc1c72313e09356f02da23ab460c857d1ac7b581b43d5f" exitCode=60 Nov 01 06:15:04.180073 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:04.180050 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891895-h46r4" event={"ID":"c6610260-baea-4921-9a3a-756fbf18eb3e","Type":"ContainerDied","Data":"97bbb6b10a65496d10cc1c72313e09356f02da23ab460c857d1ac7b581b43d5f"} Nov 01 06:15:04.180280 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:04.180261 5040 scope.go:122] "RemoveContainer" containerID="97bbb6b10a65496d10cc1c72313e09356f02da23ab460c857d1ac7b581b43d5f" Nov 01 06:15:04.180508 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:15:04.180465 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891895-h46r4_odh-ai-gateway-infra(c6610260-baea-4921-9a3a-756fbf18eb3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891895-h46r4" podUID="c6610260-baea-4921-9a3a-756fbf18eb3e" Nov 01 06:15:04.182021 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:04.181997 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891895-p4bzz" event={"ID":"0d62a50d-6bb5-4200-bc69-70deaa07d3d3","Type":"ContainerStarted","Data":"21efcd8ba4c91cbfee698c76a8fff6fd698a10de6df9b8d8aaaf6fae23565f55"} Nov 01 06:15:04.183814 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:04.183795 5040 generic.go:356] "Generic (PLEG): container finished" podID="78e5172d-422f-4082-9f96-15f1ab8b49eb" containerID="7eef78eb88c0ef378cfbd503bec531e71cb4d6badebd133490896d2f17d3ca6b" exitCode=60 Nov 01 06:15:04.183905 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:04.183851 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-w8t88" event={"ID":"78e5172d-422f-4082-9f96-15f1ab8b49eb","Type":"ContainerDied","Data":"7eef78eb88c0ef378cfbd503bec531e71cb4d6badebd133490896d2f17d3ca6b"} Nov 01 06:15:04.184040 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:04.184026 5040 scope.go:122] "RemoveContainer" containerID="7eef78eb88c0ef378cfbd503bec531e71cb4d6badebd133490896d2f17d3ca6b" Nov 01 06:15:04.184191 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:15:04.184175 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891895-w8t88_odh-ai-gateway-infra(78e5172d-422f-4082-9f96-15f1ab8b49eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-w8t88" podUID="78e5172d-422f-4082-9f96-15f1ab8b49eb" Nov 01 06:15:04.185819 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:04.185797 5040 generic.go:356] "Generic (PLEG): container finished" podID="ec119deb-0b81-447c-af06-9464177b7a01" containerID="7da4fe4587acd3e8f11974adfff2574213aa3be18cb6648b367c267ffda757d0" exitCode=60 Nov 01 06:15:04.185907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:04.185849 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec5596-29891895-nz6tn" event={"ID":"ec119deb-0b81-447c-af06-9464177b7a01","Type":"ContainerDied","Data":"7da4fe4587acd3e8f11974adfff2574213aa3be18cb6648b367c267ffda757d0"} Nov 01 06:15:04.186044 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:04.186032 5040 scope.go:122] "RemoveContainer" containerID="7da4fe4587acd3e8f11974adfff2574213aa3be18cb6648b367c267ffda757d0" Nov 01 06:15:04.186206 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:15:04.186191 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-ec5596-29891895-nz6tn_odh-ai-gateway-infra(ec119deb-0b81-447c-af06-9464177b7a01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec5596-29891895-nz6tn" podUID="ec119deb-0b81-447c-af06-9464177b7a01" Nov 01 06:15:04.187885 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:04.187866 5040 scope.go:122] "RemoveContainer" containerID="9e6799bd034721d070686b3ecb8f5393f8deb9bc84227fdc612acdb3abd4480a" Nov 01 06:15:04.261529 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:04.261505 5040 scope.go:122] "RemoveContainer" containerID="5bf7ced2675d47d23a2989acad7590ae9c48c8ea67533c58b5806f9d707bed9a" Nov 01 06:15:04.274529 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:04.274503 5040 scope.go:122] "RemoveContainer" containerID="0e608e9aed09d1c3716392695c6e2a7fd48c5243376e707facafe28b97b37097" Nov 01 06:15:04.288241 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:04.288212 5040 scope.go:122] "RemoveContainer" containerID="20908789c7e4118e1af5ecac521202e785d595bf3d68f2fbfbc9eb23cdaa8f5d" Nov 01 06:15:04.301641 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:04.301617 5040 scope.go:122] "RemoveContainer" containerID="82e29c1e5e40b607cec3101b50e1462a2ffb4c22e0c1664261ca2892403ce468" Nov 01 06:15:05.194407 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:05.194362 5040 scope.go:122] "RemoveContainer" containerID="97bbb6b10a65496d10cc1c72313e09356f02da23ab460c857d1ac7b581b43d5f" Nov 01 06:15:05.194654 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:15:05.194633 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891895-h46r4_odh-ai-gateway-infra(c6610260-baea-4921-9a3a-756fbf18eb3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891895-h46r4" podUID="c6610260-baea-4921-9a3a-756fbf18eb3e" Nov 01 06:15:05.195973 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:05.195950 5040 generic.go:356] "Generic (PLEG): container finished" podID="0d62a50d-6bb5-4200-bc69-70deaa07d3d3" containerID="21efcd8ba4c91cbfee698c76a8fff6fd698a10de6df9b8d8aaaf6fae23565f55" exitCode=60 Nov 01 06:15:05.196069 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:05.196036 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891895-p4bzz" event={"ID":"0d62a50d-6bb5-4200-bc69-70deaa07d3d3","Type":"ContainerDied","Data":"21efcd8ba4c91cbfee698c76a8fff6fd698a10de6df9b8d8aaaf6fae23565f55"} Nov 01 06:15:05.196147 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:05.196133 5040 scope.go:122] "RemoveContainer" containerID="5fb7adddac372c874f485b11f046c492bb7dae4dd2e94f96a5291242f2ef5ec2" Nov 01 06:15:05.196212 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:05.196195 5040 scope.go:122] "RemoveContainer" containerID="21efcd8ba4c91cbfee698c76a8fff6fd698a10de6df9b8d8aaaf6fae23565f55" Nov 01 06:15:05.196458 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:15:05.196428 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7d25db-29891895-p4bzz_odh-ai-gateway-infra(0d62a50d-6bb5-4200-bc69-70deaa07d3d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891895-p4bzz" podUID="0d62a50d-6bb5-4200-bc69-70deaa07d3d3" Nov 01 06:15:05.197852 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:05.197834 5040 scope.go:122] "RemoveContainer" containerID="7eef78eb88c0ef378cfbd503bec531e71cb4d6badebd133490896d2f17d3ca6b" Nov 01 06:15:05.198033 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:15:05.198016 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891895-w8t88_odh-ai-gateway-infra(78e5172d-422f-4082-9f96-15f1ab8b49eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-w8t88" podUID="78e5172d-422f-4082-9f96-15f1ab8b49eb" Nov 01 06:15:05.199263 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:05.199246 5040 scope.go:122] "RemoveContainer" containerID="7da4fe4587acd3e8f11974adfff2574213aa3be18cb6648b367c267ffda757d0" Nov 01 06:15:05.199438 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:15:05.199410 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-ec5596-29891895-nz6tn_odh-ai-gateway-infra(ec119deb-0b81-447c-af06-9464177b7a01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec5596-29891895-nz6tn" podUID="ec119deb-0b81-447c-af06-9464177b7a01" Nov 01 06:15:05.200716 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:05.200698 5040 scope.go:122] "RemoveContainer" containerID="246eaeff230ea6909723d9e9159003431958942f6fd0eb8df1e7afd756dfb7cc" Nov 01 06:15:05.200873 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:15:05.200853 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-3578ce-29891895-5vnj9_odh-ai-gateway-infra(3b1f2aca-3d31-4c8d-b83a-ca20a360e75e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3578ce-29891895-5vnj9" podUID="3b1f2aca-3d31-4c8d-b83a-ca20a360e75e" Nov 01 06:15:05.202180 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:05.202164 5040 scope.go:122] "RemoveContainer" containerID="8be3b9ae783244e7466209862e7af9903e4920b5b2220c9a79b53e10ea3f3a68" Nov 01 06:15:05.202333 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:15:05.202310 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-3986bf-29891895-mvbfb_odh-ai-gateway-infra(25021d94-5042-4ee0-b69f-12ceb0564036)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891895-mvbfb" podUID="25021d94-5042-4ee0-b69f-12ceb0564036" Nov 01 06:15:05.203706 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:05.203686 5040 scope.go:122] "RemoveContainer" containerID="7094cd180e5c51cad724f59d57a02dbaf51b5becc8b7686303e6d740aa27b26b" Nov 01 06:15:05.203879 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:15:05.203860 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891895-bb2zn_odh-ai-gateway-infra(7b7f1411-aaf9-4d3b-91c0-40f875416c57)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891895-bb2zn" podUID="7b7f1411-aaf9-4d3b-91c0-40f875416c57" Nov 01 06:15:06.211637 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:06.211608 5040 scope.go:122] "RemoveContainer" containerID="21efcd8ba4c91cbfee698c76a8fff6fd698a10de6df9b8d8aaaf6fae23565f55" Nov 01 06:15:06.212019 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:15:06.211865 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7d25db-29891895-p4bzz_odh-ai-gateway-infra(0d62a50d-6bb5-4200-bc69-70deaa07d3d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891895-p4bzz" podUID="0d62a50d-6bb5-4200-bc69-70deaa07d3d3" Nov 01 06:15:09.973574 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:15:09.973538 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:15:10.047397 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:10.047359 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ead5c4a-fnffv"] Nov 01 06:15:10.097920 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:10.097881 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ead5c4a-fnffv"] Nov 01 06:15:11.515792 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:11.515760 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e993ed84-a606-4616-b122-3319a82c5633" path="/var/lib/kubelet/pods/e993ed84-a606-4616-b122-3319a82c5633/volumes" Nov 01 06:15:14.508293 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:14.508263 5040 scope.go:122] "RemoveContainer" containerID="7da4fe4587acd3e8f11974adfff2574213aa3be18cb6648b367c267ffda757d0" Nov 01 06:15:14.508811 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:14.508376 5040 scope.go:122] "RemoveContainer" containerID="8be3b9ae783244e7466209862e7af9903e4920b5b2220c9a79b53e10ea3f3a68" Nov 01 06:15:14.508811 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:14.508466 5040 scope.go:122] "RemoveContainer" containerID="246eaeff230ea6909723d9e9159003431958942f6fd0eb8df1e7afd756dfb7cc" Nov 01 06:15:14.508811 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:14.508560 5040 scope.go:122] "RemoveContainer" containerID="7094cd180e5c51cad724f59d57a02dbaf51b5becc8b7686303e6d740aa27b26b" Nov 01 06:15:15.270394 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:15.270356 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891895-mvbfb" event={"ID":"25021d94-5042-4ee0-b69f-12ceb0564036","Type":"ContainerStarted","Data":"c92a4a98bf3128fcc7c9f1c6becaf91de08f44a5e5f9a6b601293d7cd0fdd7b1"} Nov 01 06:15:15.272305 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:15.272280 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891895-bb2zn" event={"ID":"7b7f1411-aaf9-4d3b-91c0-40f875416c57","Type":"ContainerStarted","Data":"cee3498e62b2545ac8429f22d3422d01b38529d4dd75fa511a8c4add11f6a6ab"} Nov 01 06:15:15.274164 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:15.274141 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec5596-29891895-nz6tn" event={"ID":"ec119deb-0b81-447c-af06-9464177b7a01","Type":"ContainerStarted","Data":"e2066d2177c3b35a6b400a4e05893caab7603d4e99f3b3bfc09975858aac78fc"} Nov 01 06:15:15.276042 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:15.276022 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3578ce-29891895-5vnj9" event={"ID":"3b1f2aca-3d31-4c8d-b83a-ca20a360e75e","Type":"ContainerStarted","Data":"8e35ef59384ed8b4ef5f69c61ae702977373aa79baf020ed753020cf2a35637d"} Nov 01 06:15:15.507674 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:15.507649 5040 scope.go:122] "RemoveContainer" containerID="97bbb6b10a65496d10cc1c72313e09356f02da23ab460c857d1ac7b581b43d5f" Nov 01 06:15:15.507819 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:15.507800 5040 scope.go:122] "RemoveContainer" containerID="21efcd8ba4c91cbfee698c76a8fff6fd698a10de6df9b8d8aaaf6fae23565f55" Nov 01 06:15:16.287832 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:16.287790 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891895-p4bzz" event={"ID":"0d62a50d-6bb5-4200-bc69-70deaa07d3d3","Type":"ContainerStarted","Data":"90ca8867c251d0aa5ae6ca73a2a5c31bb4ef06322a0ca9d587746ee82e21ef65"} Nov 01 06:15:16.289726 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:16.289704 5040 generic.go:356] "Generic (PLEG): container finished" podID="ec119deb-0b81-447c-af06-9464177b7a01" containerID="e2066d2177c3b35a6b400a4e05893caab7603d4e99f3b3bfc09975858aac78fc" exitCode=60 Nov 01 06:15:16.289843 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:16.289778 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec5596-29891895-nz6tn" event={"ID":"ec119deb-0b81-447c-af06-9464177b7a01","Type":"ContainerDied","Data":"e2066d2177c3b35a6b400a4e05893caab7603d4e99f3b3bfc09975858aac78fc"} Nov 01 06:15:16.289843 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:16.289834 5040 scope.go:122] "RemoveContainer" containerID="7da4fe4587acd3e8f11974adfff2574213aa3be18cb6648b367c267ffda757d0" Nov 01 06:15:16.290131 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:16.290109 5040 scope.go:122] "RemoveContainer" containerID="e2066d2177c3b35a6b400a4e05893caab7603d4e99f3b3bfc09975858aac78fc" Nov 01 06:15:16.290357 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:15:16.290336 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-ec5596-29891895-nz6tn_odh-ai-gateway-infra(ec119deb-0b81-447c-af06-9464177b7a01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec5596-29891895-nz6tn" podUID="ec119deb-0b81-447c-af06-9464177b7a01" Nov 01 06:15:16.291721 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:16.291703 5040 generic.go:356] "Generic (PLEG): container finished" podID="3b1f2aca-3d31-4c8d-b83a-ca20a360e75e" containerID="8e35ef59384ed8b4ef5f69c61ae702977373aa79baf020ed753020cf2a35637d" exitCode=60 Nov 01 06:15:16.291800 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:16.291775 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3578ce-29891895-5vnj9" event={"ID":"3b1f2aca-3d31-4c8d-b83a-ca20a360e75e","Type":"ContainerDied","Data":"8e35ef59384ed8b4ef5f69c61ae702977373aa79baf020ed753020cf2a35637d"} Nov 01 06:15:16.292076 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:16.292060 5040 scope.go:122] "RemoveContainer" containerID="8e35ef59384ed8b4ef5f69c61ae702977373aa79baf020ed753020cf2a35637d" Nov 01 06:15:16.292272 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:15:16.292253 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-3578ce-29891895-5vnj9_odh-ai-gateway-infra(3b1f2aca-3d31-4c8d-b83a-ca20a360e75e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3578ce-29891895-5vnj9" podUID="3b1f2aca-3d31-4c8d-b83a-ca20a360e75e" Nov 01 06:15:16.293652 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:16.293628 5040 generic.go:356] "Generic (PLEG): container finished" podID="25021d94-5042-4ee0-b69f-12ceb0564036" containerID="c92a4a98bf3128fcc7c9f1c6becaf91de08f44a5e5f9a6b601293d7cd0fdd7b1" exitCode=60 Nov 01 06:15:16.293744 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:16.293701 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891895-mvbfb" event={"ID":"25021d94-5042-4ee0-b69f-12ceb0564036","Type":"ContainerDied","Data":"c92a4a98bf3128fcc7c9f1c6becaf91de08f44a5e5f9a6b601293d7cd0fdd7b1"} Nov 01 06:15:16.294084 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:16.294069 5040 scope.go:122] "RemoveContainer" containerID="c92a4a98bf3128fcc7c9f1c6becaf91de08f44a5e5f9a6b601293d7cd0fdd7b1" Nov 01 06:15:16.294284 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:15:16.294267 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-3986bf-29891895-mvbfb_odh-ai-gateway-infra(25021d94-5042-4ee0-b69f-12ceb0564036)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891895-mvbfb" podUID="25021d94-5042-4ee0-b69f-12ceb0564036" Nov 01 06:15:16.295626 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:16.295604 5040 generic.go:356] "Generic (PLEG): container finished" podID="7b7f1411-aaf9-4d3b-91c0-40f875416c57" containerID="cee3498e62b2545ac8429f22d3422d01b38529d4dd75fa511a8c4add11f6a6ab" exitCode=60 Nov 01 06:15:16.295710 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:16.295671 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891895-bb2zn" event={"ID":"7b7f1411-aaf9-4d3b-91c0-40f875416c57","Type":"ContainerDied","Data":"cee3498e62b2545ac8429f22d3422d01b38529d4dd75fa511a8c4add11f6a6ab"} Nov 01 06:15:16.295959 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:16.295944 5040 scope.go:122] "RemoveContainer" containerID="cee3498e62b2545ac8429f22d3422d01b38529d4dd75fa511a8c4add11f6a6ab" Nov 01 06:15:16.296152 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:15:16.296133 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891895-bb2zn_odh-ai-gateway-infra(7b7f1411-aaf9-4d3b-91c0-40f875416c57)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891895-bb2zn" podUID="7b7f1411-aaf9-4d3b-91c0-40f875416c57" Nov 01 06:15:16.297503 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:16.297473 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891895-h46r4" event={"ID":"c6610260-baea-4921-9a3a-756fbf18eb3e","Type":"ContainerStarted","Data":"16c6adb52cc9adb942703d1d6be65f6e0767b037f5f5b577f9e7dfa864efa8a2"} Nov 01 06:15:16.303262 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:16.303246 5040 scope.go:122] "RemoveContainer" containerID="246eaeff230ea6909723d9e9159003431958942f6fd0eb8df1e7afd756dfb7cc" Nov 01 06:15:16.315625 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:16.315606 5040 scope.go:122] "RemoveContainer" containerID="8be3b9ae783244e7466209862e7af9903e4920b5b2220c9a79b53e10ea3f3a68" Nov 01 06:15:16.327520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:16.327501 5040 scope.go:122] "RemoveContainer" containerID="7094cd180e5c51cad724f59d57a02dbaf51b5becc8b7686303e6d740aa27b26b" Nov 01 06:15:16.397529 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:16.397498 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891895-mvbfb"] Nov 01 06:15:16.507431 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:16.507398 5040 scope.go:122] "RemoveContainer" containerID="7eef78eb88c0ef378cfbd503bec531e71cb4d6badebd133490896d2f17d3ca6b" Nov 01 06:15:16.513794 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:16.513769 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec5596-29891895-nz6tn"] Nov 01 06:15:16.629062 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:16.629030 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3578ce-29891895-5vnj9"] Nov 01 06:15:16.744164 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:16.744133 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891895-bb2zn"] Nov 01 06:15:17.317240 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.317210 5040 generic.go:356] "Generic (PLEG): container finished" podID="c6610260-baea-4921-9a3a-756fbf18eb3e" containerID="16c6adb52cc9adb942703d1d6be65f6e0767b037f5f5b577f9e7dfa864efa8a2" exitCode=60 Nov 01 06:15:17.317671 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.317297 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891895-h46r4" event={"ID":"c6610260-baea-4921-9a3a-756fbf18eb3e","Type":"ContainerDied","Data":"16c6adb52cc9adb942703d1d6be65f6e0767b037f5f5b577f9e7dfa864efa8a2"} Nov 01 06:15:17.317671 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.317400 5040 scope.go:122] "RemoveContainer" containerID="97bbb6b10a65496d10cc1c72313e09356f02da23ab460c857d1ac7b581b43d5f" Nov 01 06:15:17.317935 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.317910 5040 scope.go:122] "RemoveContainer" containerID="16c6adb52cc9adb942703d1d6be65f6e0767b037f5f5b577f9e7dfa864efa8a2" Nov 01 06:15:17.318221 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:15:17.318192 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891895-h46r4_odh-ai-gateway-infra(c6610260-baea-4921-9a3a-756fbf18eb3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891895-h46r4" podUID="c6610260-baea-4921-9a3a-756fbf18eb3e" Nov 01 06:15:17.320377 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.320349 5040 generic.go:356] "Generic (PLEG): container finished" podID="0d62a50d-6bb5-4200-bc69-70deaa07d3d3" containerID="90ca8867c251d0aa5ae6ca73a2a5c31bb4ef06322a0ca9d587746ee82e21ef65" exitCode=60 Nov 01 06:15:17.320518 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.320444 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891895-p4bzz" event={"ID":"0d62a50d-6bb5-4200-bc69-70deaa07d3d3","Type":"ContainerDied","Data":"90ca8867c251d0aa5ae6ca73a2a5c31bb4ef06322a0ca9d587746ee82e21ef65"} Nov 01 06:15:17.321041 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.320813 5040 scope.go:122] "RemoveContainer" containerID="90ca8867c251d0aa5ae6ca73a2a5c31bb4ef06322a0ca9d587746ee82e21ef65" Nov 01 06:15:17.321152 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:15:17.321055 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7d25db-29891895-p4bzz_odh-ai-gateway-infra(0d62a50d-6bb5-4200-bc69-70deaa07d3d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891895-p4bzz" podUID="0d62a50d-6bb5-4200-bc69-70deaa07d3d3" Nov 01 06:15:17.324570 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.323996 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-w8t88" event={"ID":"78e5172d-422f-4082-9f96-15f1ab8b49eb","Type":"ContainerStarted","Data":"1cde6b7767c721a564bf27b419d18689d9da970ff302d5b05563b2de4e662438"} Nov 01 06:15:17.335674 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.335305 5040 scope.go:122] "RemoveContainer" containerID="21efcd8ba4c91cbfee698c76a8fff6fd698a10de6df9b8d8aaaf6fae23565f55" Nov 01 06:15:17.418860 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.418657 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891895-p4bzz"] Nov 01 06:15:17.515315 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.515288 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891895-mvbfb" Nov 01 06:15:17.575395 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.575329 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891895-bb2zn" Nov 01 06:15:17.623149 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.623117 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25021d94-5042-4ee0-b69f-12ceb0564036-kube-api-access-d9bjq\" (UniqueName: \"kubernetes.io/projected/25021d94-5042-4ee0-b69f-12ceb0564036-kube-api-access-d9bjq\") pod \"25021d94-5042-4ee0-b69f-12ceb0564036\" (UID: \"25021d94-5042-4ee0-b69f-12ceb0564036\") " Nov 01 06:15:17.623326 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.623211 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25021d94-5042-4ee0-b69f-12ceb0564036-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25021d94-5042-4ee0-b69f-12ceb0564036-maas-api-service-ca\") pod \"25021d94-5042-4ee0-b69f-12ceb0564036\" (UID: \"25021d94-5042-4ee0-b69f-12ceb0564036\") " Nov 01 06:15:17.623326 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.623295 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b7f1411-aaf9-4d3b-91c0-40f875416c57-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b7f1411-aaf9-4d3b-91c0-40f875416c57-maas-api-service-ca\") pod \"7b7f1411-aaf9-4d3b-91c0-40f875416c57\" (UID: \"7b7f1411-aaf9-4d3b-91c0-40f875416c57\") " Nov 01 06:15:17.623450 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.623389 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b7f1411-aaf9-4d3b-91c0-40f875416c57-kube-api-access-kb856\" (UniqueName: \"kubernetes.io/projected/7b7f1411-aaf9-4d3b-91c0-40f875416c57-kube-api-access-kb856\") pod \"7b7f1411-aaf9-4d3b-91c0-40f875416c57\" (UID: \"7b7f1411-aaf9-4d3b-91c0-40f875416c57\") " Nov 01 06:15:17.624716 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.624683 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25021d94-5042-4ee0-b69f-12ceb0564036-maas-api-service-ca" pod "25021d94-5042-4ee0-b69f-12ceb0564036" (UID: "25021d94-5042-4ee0-b69f-12ceb0564036"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:15:17.626473 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.626446 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25021d94-5042-4ee0-b69f-12ceb0564036-kube-api-access-d9bjq" pod "25021d94-5042-4ee0-b69f-12ceb0564036" (UID: "25021d94-5042-4ee0-b69f-12ceb0564036"). InnerVolumeSpecName "kube-api-access-d9bjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:15:17.627523 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.627213 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b7f1411-aaf9-4d3b-91c0-40f875416c57-maas-api-service-ca" pod "7b7f1411-aaf9-4d3b-91c0-40f875416c57" (UID: "7b7f1411-aaf9-4d3b-91c0-40f875416c57"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:15:17.627765 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.627745 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b7f1411-aaf9-4d3b-91c0-40f875416c57-kube-api-access-kb856" pod "7b7f1411-aaf9-4d3b-91c0-40f875416c57" (UID: "7b7f1411-aaf9-4d3b-91c0-40f875416c57"). InnerVolumeSpecName "kube-api-access-kb856". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:15:17.647339 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.647255 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3578ce-29891895-5vnj9" Nov 01 06:15:17.653378 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.653353 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec5596-29891895-nz6tn" Nov 01 06:15:17.725455 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.725424 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b1f2aca-3d31-4c8d-b83a-ca20a360e75e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1f2aca-3d31-4c8d-b83a-ca20a360e75e-maas-api-service-ca\") pod \"3b1f2aca-3d31-4c8d-b83a-ca20a360e75e\" (UID: \"3b1f2aca-3d31-4c8d-b83a-ca20a360e75e\") " Nov 01 06:15:17.725778 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.725533 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec119deb-0b81-447c-af06-9464177b7a01-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec119deb-0b81-447c-af06-9464177b7a01-maas-api-service-ca\") pod \"ec119deb-0b81-447c-af06-9464177b7a01\" (UID: \"ec119deb-0b81-447c-af06-9464177b7a01\") " Nov 01 06:15:17.725778 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.725565 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b1f2aca-3d31-4c8d-b83a-ca20a360e75e-kube-api-access-4f7p8\" (UniqueName: \"kubernetes.io/projected/3b1f2aca-3d31-4c8d-b83a-ca20a360e75e-kube-api-access-4f7p8\") pod \"3b1f2aca-3d31-4c8d-b83a-ca20a360e75e\" (UID: \"3b1f2aca-3d31-4c8d-b83a-ca20a360e75e\") " Nov 01 06:15:17.725778 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.725653 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec119deb-0b81-447c-af06-9464177b7a01-kube-api-access-sjvr8\" (UniqueName: \"kubernetes.io/projected/ec119deb-0b81-447c-af06-9464177b7a01-kube-api-access-sjvr8\") pod \"ec119deb-0b81-447c-af06-9464177b7a01\" (UID: \"ec119deb-0b81-447c-af06-9464177b7a01\") " Nov 01 06:15:17.725935 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.725837 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b1f2aca-3d31-4c8d-b83a-ca20a360e75e-maas-api-service-ca" pod "3b1f2aca-3d31-4c8d-b83a-ca20a360e75e" (UID: "3b1f2aca-3d31-4c8d-b83a-ca20a360e75e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:15:17.725975 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.725925 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec119deb-0b81-447c-af06-9464177b7a01-maas-api-service-ca" pod "ec119deb-0b81-447c-af06-9464177b7a01" (UID: "ec119deb-0b81-447c-af06-9464177b7a01"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:15:17.726077 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.726059 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b7f1411-aaf9-4d3b-91c0-40f875416c57-maas-api-service-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:17.726118 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.726082 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb856\" (UniqueName: \"kubernetes.io/projected/7b7f1411-aaf9-4d3b-91c0-40f875416c57-kube-api-access-kb856\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:17.726118 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.726099 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1f2aca-3d31-4c8d-b83a-ca20a360e75e-maas-api-service-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:17.726183 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.726113 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9bjq\" (UniqueName: \"kubernetes.io/projected/25021d94-5042-4ee0-b69f-12ceb0564036-kube-api-access-d9bjq\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:17.726183 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.726128 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec119deb-0b81-447c-af06-9464177b7a01-maas-api-service-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:17.726183 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.726142 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25021d94-5042-4ee0-b69f-12ceb0564036-maas-api-service-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:17.727813 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.727781 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec119deb-0b81-447c-af06-9464177b7a01-kube-api-access-sjvr8" pod "ec119deb-0b81-447c-af06-9464177b7a01" (UID: "ec119deb-0b81-447c-af06-9464177b7a01"). InnerVolumeSpecName "kube-api-access-sjvr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:15:17.727813 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.727802 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b1f2aca-3d31-4c8d-b83a-ca20a360e75e-kube-api-access-4f7p8" pod "3b1f2aca-3d31-4c8d-b83a-ca20a360e75e" (UID: "3b1f2aca-3d31-4c8d-b83a-ca20a360e75e"). InnerVolumeSpecName "kube-api-access-4f7p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:15:17.827127 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.827043 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f7p8\" (UniqueName: \"kubernetes.io/projected/3b1f2aca-3d31-4c8d-b83a-ca20a360e75e-kube-api-access-4f7p8\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:17.827127 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.827080 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjvr8\" (UniqueName: \"kubernetes.io/projected/ec119deb-0b81-447c-af06-9464177b7a01-kube-api-access-sjvr8\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:17.887252 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:17.887212 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891895-h46r4"] Nov 01 06:15:18.337002 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:18.336964 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891895-mvbfb" event={"ID":"25021d94-5042-4ee0-b69f-12ceb0564036","Type":"ContainerDied","Data":"b47f50ebf503f9e991fcb567e7eef8cbada270a5041775b0e655e4c9b06082d5"} Nov 01 06:15:18.337529 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:18.336983 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891895-mvbfb" Nov 01 06:15:18.337529 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:18.337034 5040 scope.go:122] "RemoveContainer" containerID="c92a4a98bf3128fcc7c9f1c6becaf91de08f44a5e5f9a6b601293d7cd0fdd7b1" Nov 01 06:15:18.338543 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:18.338520 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891895-bb2zn" event={"ID":"7b7f1411-aaf9-4d3b-91c0-40f875416c57","Type":"ContainerDied","Data":"1b2874a77a1123cde474f9652b628c36bd7efa87a9705266e2c33a712c15f9d5"} Nov 01 06:15:18.338717 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:18.338533 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891895-bb2zn" Nov 01 06:15:18.350657 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:18.350632 5040 generic.go:356] "Generic (PLEG): container finished" podID="78e5172d-422f-4082-9f96-15f1ab8b49eb" containerID="1cde6b7767c721a564bf27b419d18689d9da970ff302d5b05563b2de4e662438" exitCode=60 Nov 01 06:15:18.350769 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:18.350686 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-w8t88" event={"ID":"78e5172d-422f-4082-9f96-15f1ab8b49eb","Type":"ContainerDied","Data":"1cde6b7767c721a564bf27b419d18689d9da970ff302d5b05563b2de4e662438"} Nov 01 06:15:18.351029 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:18.351012 5040 scope.go:122] "RemoveContainer" containerID="1cde6b7767c721a564bf27b419d18689d9da970ff302d5b05563b2de4e662438" Nov 01 06:15:18.351272 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:15:18.351252 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891895-w8t88_odh-ai-gateway-infra(78e5172d-422f-4082-9f96-15f1ab8b49eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-w8t88" podUID="78e5172d-422f-4082-9f96-15f1ab8b49eb" Nov 01 06:15:18.352933 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:18.352908 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec5596-29891895-nz6tn" Nov 01 06:15:18.353041 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:18.352942 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec5596-29891895-nz6tn" event={"ID":"ec119deb-0b81-447c-af06-9464177b7a01","Type":"ContainerDied","Data":"21e4a8a8097d63fbad5b834d6092f460b7fb059c4d8d3dc3b392f5dfa361c739"} Nov 01 06:15:18.353363 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:18.353333 5040 scope.go:122] "RemoveContainer" containerID="cee3498e62b2545ac8429f22d3422d01b38529d4dd75fa511a8c4add11f6a6ab" Nov 01 06:15:18.355518 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:18.354841 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3578ce-29891895-5vnj9" Nov 01 06:15:18.355518 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:18.354871 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3578ce-29891895-5vnj9" event={"ID":"3b1f2aca-3d31-4c8d-b83a-ca20a360e75e","Type":"ContainerDied","Data":"8009e9cb0a652f927d8c05915234544a629820475fdb0b2d5ca9fb19203193bb"} Nov 01 06:15:18.366325 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:18.366302 5040 scope.go:122] "RemoveContainer" containerID="7eef78eb88c0ef378cfbd503bec531e71cb4d6badebd133490896d2f17d3ca6b" Nov 01 06:15:18.386343 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:18.385988 5040 scope.go:122] "RemoveContainer" containerID="e2066d2177c3b35a6b400a4e05893caab7603d4e99f3b3bfc09975858aac78fc" Nov 01 06:15:18.407284 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:18.407209 5040 scope.go:122] "RemoveContainer" containerID="8e35ef59384ed8b4ef5f69c61ae702977373aa79baf020ed753020cf2a35637d" Nov 01 06:15:18.530690 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:18.530663 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891895-p4bzz" Nov 01 06:15:18.535757 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:18.535737 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891895-h46r4" Nov 01 06:15:18.634836 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:18.634798 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d62a50d-6bb5-4200-bc69-70deaa07d3d3-kube-api-access-pwstr\" (UniqueName: \"kubernetes.io/projected/0d62a50d-6bb5-4200-bc69-70deaa07d3d3-kube-api-access-pwstr\") pod \"0d62a50d-6bb5-4200-bc69-70deaa07d3d3\" (UID: \"0d62a50d-6bb5-4200-bc69-70deaa07d3d3\") " Nov 01 06:15:18.635006 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:18.634857 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6610260-baea-4921-9a3a-756fbf18eb3e-kube-api-access-m8dv2\" (UniqueName: \"kubernetes.io/projected/c6610260-baea-4921-9a3a-756fbf18eb3e-kube-api-access-m8dv2\") pod \"c6610260-baea-4921-9a3a-756fbf18eb3e\" (UID: \"c6610260-baea-4921-9a3a-756fbf18eb3e\") " Nov 01 06:15:18.635006 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:18.634970 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d62a50d-6bb5-4200-bc69-70deaa07d3d3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d62a50d-6bb5-4200-bc69-70deaa07d3d3-maas-api-service-ca\") pod \"0d62a50d-6bb5-4200-bc69-70deaa07d3d3\" (UID: \"0d62a50d-6bb5-4200-bc69-70deaa07d3d3\") " Nov 01 06:15:18.635139 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:18.635046 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6610260-baea-4921-9a3a-756fbf18eb3e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6610260-baea-4921-9a3a-756fbf18eb3e-maas-api-service-ca\") pod \"c6610260-baea-4921-9a3a-756fbf18eb3e\" (UID: \"c6610260-baea-4921-9a3a-756fbf18eb3e\") " Nov 01 06:15:18.635620 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:18.635563 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6610260-baea-4921-9a3a-756fbf18eb3e-maas-api-service-ca" pod "c6610260-baea-4921-9a3a-756fbf18eb3e" (UID: "c6610260-baea-4921-9a3a-756fbf18eb3e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:15:18.636102 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:18.636076 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d62a50d-6bb5-4200-bc69-70deaa07d3d3-maas-api-service-ca" pod "0d62a50d-6bb5-4200-bc69-70deaa07d3d3" (UID: "0d62a50d-6bb5-4200-bc69-70deaa07d3d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:15:18.637190 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:18.637169 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6610260-baea-4921-9a3a-756fbf18eb3e-kube-api-access-m8dv2" pod "c6610260-baea-4921-9a3a-756fbf18eb3e" (UID: "c6610260-baea-4921-9a3a-756fbf18eb3e"). InnerVolumeSpecName "kube-api-access-m8dv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:15:18.637288 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:18.637183 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d62a50d-6bb5-4200-bc69-70deaa07d3d3-kube-api-access-pwstr" pod "0d62a50d-6bb5-4200-bc69-70deaa07d3d3" (UID: "0d62a50d-6bb5-4200-bc69-70deaa07d3d3"). InnerVolumeSpecName "kube-api-access-pwstr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:15:18.679828 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:18.679801 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891895-w8t88"] Nov 01 06:15:18.717342 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:18.717310 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891895-bb2zn"] Nov 01 06:15:18.723785 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:18.723757 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891895-bb2zn"] Nov 01 06:15:18.736223 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:18.736197 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwstr\" (UniqueName: \"kubernetes.io/projected/0d62a50d-6bb5-4200-bc69-70deaa07d3d3-kube-api-access-pwstr\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:18.736223 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:18.736223 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8dv2\" (UniqueName: \"kubernetes.io/projected/c6610260-baea-4921-9a3a-756fbf18eb3e-kube-api-access-m8dv2\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:18.736383 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:18.736233 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d62a50d-6bb5-4200-bc69-70deaa07d3d3-maas-api-service-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:18.736383 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:18.736252 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6610260-baea-4921-9a3a-756fbf18eb3e-maas-api-service-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:18.950056 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:18.949960 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec5596-29891895-nz6tn"] Nov 01 06:15:18.953412 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:18.953386 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ec5596-29891895-nz6tn"] Nov 01 06:15:19.081516 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.081460 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e004500a-tmq5k"] Nov 01 06:15:19.082177 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082155 5040 cpu_manager.go:414] "containerMap: removing container" podUID="25021d94-5042-4ee0-b69f-12ceb0564036" containerName="cleanup" Nov 01 06:15:19.082229 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082181 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25021d94-5042-4ee0-b69f-12ceb0564036" containerName="cleanup" Nov 01 06:15:19.082229 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082195 5040 cpu_manager.go:414] "containerMap: removing container" podUID="25021d94-5042-4ee0-b69f-12ceb0564036" containerName="cleanup" Nov 01 06:15:19.082229 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082202 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25021d94-5042-4ee0-b69f-12ceb0564036" containerName="cleanup" Nov 01 06:15:19.082229 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082218 5040 cpu_manager.go:414] "containerMap: removing container" podUID="ec119deb-0b81-447c-af06-9464177b7a01" containerName="cleanup" Nov 01 06:15:19.082229 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082224 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec119deb-0b81-447c-af06-9464177b7a01" containerName="cleanup" Nov 01 06:15:19.082388 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082237 5040 cpu_manager.go:414] "containerMap: removing container" podUID="3b1f2aca-3d31-4c8d-b83a-ca20a360e75e" containerName="cleanup" Nov 01 06:15:19.082388 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082243 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b1f2aca-3d31-4c8d-b83a-ca20a360e75e" containerName="cleanup" Nov 01 06:15:19.082388 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082258 5040 cpu_manager.go:414] "containerMap: removing container" podUID="ec119deb-0b81-447c-af06-9464177b7a01" containerName="cleanup" Nov 01 06:15:19.082388 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082264 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec119deb-0b81-447c-af06-9464177b7a01" containerName="cleanup" Nov 01 06:15:19.082388 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082279 5040 cpu_manager.go:414] "containerMap: removing container" podUID="7b7f1411-aaf9-4d3b-91c0-40f875416c57" containerName="cleanup" Nov 01 06:15:19.082388 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082285 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b7f1411-aaf9-4d3b-91c0-40f875416c57" containerName="cleanup" Nov 01 06:15:19.082388 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082295 5040 cpu_manager.go:414] "containerMap: removing container" podUID="0d62a50d-6bb5-4200-bc69-70deaa07d3d3" containerName="cleanup" Nov 01 06:15:19.082388 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082302 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d62a50d-6bb5-4200-bc69-70deaa07d3d3" containerName="cleanup" Nov 01 06:15:19.082388 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082311 5040 cpu_manager.go:414] "containerMap: removing container" podUID="ec119deb-0b81-447c-af06-9464177b7a01" containerName="cleanup" Nov 01 06:15:19.082388 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082317 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec119deb-0b81-447c-af06-9464177b7a01" containerName="cleanup" Nov 01 06:15:19.082388 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082347 5040 cpu_manager.go:414] "containerMap: removing container" podUID="3b1f2aca-3d31-4c8d-b83a-ca20a360e75e" containerName="cleanup" Nov 01 06:15:19.082388 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082353 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b1f2aca-3d31-4c8d-b83a-ca20a360e75e" containerName="cleanup" Nov 01 06:15:19.082388 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082363 5040 cpu_manager.go:414] "containerMap: removing container" podUID="c6610260-baea-4921-9a3a-756fbf18eb3e" containerName="cleanup" Nov 01 06:15:19.082388 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082370 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6610260-baea-4921-9a3a-756fbf18eb3e" containerName="cleanup" Nov 01 06:15:19.082388 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082386 5040 cpu_manager.go:414] "containerMap: removing container" podUID="c6610260-baea-4921-9a3a-756fbf18eb3e" containerName="cleanup" Nov 01 06:15:19.082388 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082393 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6610260-baea-4921-9a3a-756fbf18eb3e" containerName="cleanup" Nov 01 06:15:19.082904 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082426 5040 cpu_manager.go:414] "containerMap: removing container" podUID="0d62a50d-6bb5-4200-bc69-70deaa07d3d3" containerName="cleanup" Nov 01 06:15:19.082904 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082433 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d62a50d-6bb5-4200-bc69-70deaa07d3d3" containerName="cleanup" Nov 01 06:15:19.082904 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082449 5040 cpu_manager.go:414] "containerMap: removing container" podUID="7b7f1411-aaf9-4d3b-91c0-40f875416c57" containerName="cleanup" Nov 01 06:15:19.082904 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082455 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b7f1411-aaf9-4d3b-91c0-40f875416c57" containerName="cleanup" Nov 01 06:15:19.082904 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082617 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec119deb-0b81-447c-af06-9464177b7a01" containerName="cleanup" Nov 01 06:15:19.082904 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082631 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b7f1411-aaf9-4d3b-91c0-40f875416c57" containerName="cleanup" Nov 01 06:15:19.082904 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082639 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d62a50d-6bb5-4200-bc69-70deaa07d3d3" containerName="cleanup" Nov 01 06:15:19.082904 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082647 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec119deb-0b81-447c-af06-9464177b7a01" containerName="cleanup" Nov 01 06:15:19.082904 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082657 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b1f2aca-3d31-4c8d-b83a-ca20a360e75e" containerName="cleanup" Nov 01 06:15:19.082904 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082666 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b1f2aca-3d31-4c8d-b83a-ca20a360e75e" containerName="cleanup" Nov 01 06:15:19.082904 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082676 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6610260-baea-4921-9a3a-756fbf18eb3e" containerName="cleanup" Nov 01 06:15:19.082904 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082685 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="25021d94-5042-4ee0-b69f-12ceb0564036" containerName="cleanup" Nov 01 06:15:19.082904 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082693 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6610260-baea-4921-9a3a-756fbf18eb3e" containerName="cleanup" Nov 01 06:15:19.082904 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082704 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b7f1411-aaf9-4d3b-91c0-40f875416c57" containerName="cleanup" Nov 01 06:15:19.082904 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082712 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d62a50d-6bb5-4200-bc69-70deaa07d3d3" containerName="cleanup" Nov 01 06:15:19.082904 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082720 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="25021d94-5042-4ee0-b69f-12ceb0564036" containerName="cleanup" Nov 01 06:15:19.082904 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082729 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="25021d94-5042-4ee0-b69f-12ceb0564036" containerName="cleanup" Nov 01 06:15:19.082904 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.082738 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b1f2aca-3d31-4c8d-b83a-ca20a360e75e" containerName="cleanup" Nov 01 06:15:19.089020 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.088998 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e004500a-tmq5k" Nov 01 06:15:19.140640 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.140603 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jmvx\" (UniqueName: \"kubernetes.io/projected/89cda59d-10d7-434f-890e-6e5c5b1caf45-kube-api-access-8jmvx\") pod \"maas-api-revoke-sub-e004500a-tmq5k\" (UID: \"89cda59d-10d7-434f-890e-6e5c5b1caf45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e004500a-tmq5k" Nov 01 06:15:19.140827 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.140783 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89cda59d-10d7-434f-890e-6e5c5b1caf45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e004500a-tmq5k\" (UID: \"89cda59d-10d7-434f-890e-6e5c5b1caf45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e004500a-tmq5k" Nov 01 06:15:19.140909 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.140835 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89cda59d-10d7-434f-890e-6e5c5b1caf45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e004500a-tmq5k\" (UID: \"89cda59d-10d7-434f-890e-6e5c5b1caf45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e004500a-tmq5k" Nov 01 06:15:19.141815 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.141797 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:15:19.142398 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.142379 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e004500a-tmq5k"] Nov 01 06:15:19.181572 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.181543 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891895-mvbfb"] Nov 01 06:15:19.186699 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.186668 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891895-mvbfb"] Nov 01 06:15:19.242075 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.241989 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89cda59d-10d7-434f-890e-6e5c5b1caf45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e004500a-tmq5k\" (UID: \"89cda59d-10d7-434f-890e-6e5c5b1caf45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e004500a-tmq5k" Nov 01 06:15:19.242075 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.242031 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89cda59d-10d7-434f-890e-6e5c5b1caf45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e004500a-tmq5k\" (UID: \"89cda59d-10d7-434f-890e-6e5c5b1caf45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e004500a-tmq5k" Nov 01 06:15:19.242276 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.242096 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jmvx\" (UniqueName: \"kubernetes.io/projected/89cda59d-10d7-434f-890e-6e5c5b1caf45-kube-api-access-8jmvx\") pod \"maas-api-revoke-sub-e004500a-tmq5k\" (UID: \"89cda59d-10d7-434f-890e-6e5c5b1caf45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e004500a-tmq5k" Nov 01 06:15:19.242888 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.242863 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89cda59d-10d7-434f-890e-6e5c5b1caf45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e004500a-tmq5k\" (UID: \"89cda59d-10d7-434f-890e-6e5c5b1caf45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e004500a-tmq5k" Nov 01 06:15:19.244387 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.244366 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89cda59d-10d7-434f-890e-6e5c5b1caf45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e004500a-tmq5k\" (UID: \"89cda59d-10d7-434f-890e-6e5c5b1caf45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e004500a-tmq5k" Nov 01 06:15:19.307396 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.307365 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jmvx\" (UniqueName: \"kubernetes.io/projected/89cda59d-10d7-434f-890e-6e5c5b1caf45-kube-api-access-8jmvx\") pod \"maas-api-revoke-sub-e004500a-tmq5k\" (UID: \"89cda59d-10d7-434f-890e-6e5c5b1caf45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e004500a-tmq5k" Nov 01 06:15:19.367342 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.367302 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891895-h46r4" Nov 01 06:15:19.367342 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.367334 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891895-h46r4" event={"ID":"c6610260-baea-4921-9a3a-756fbf18eb3e","Type":"ContainerDied","Data":"e91db8397a0f833cdcc4d67e220502b2b4981a8c1a00db524e6c20de8d00fc8e"} Nov 01 06:15:19.367920 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.367402 5040 scope.go:122] "RemoveContainer" containerID="16c6adb52cc9adb942703d1d6be65f6e0767b037f5f5b577f9e7dfa864efa8a2" Nov 01 06:15:19.369040 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.368914 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891895-p4bzz" Nov 01 06:15:19.369040 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.368926 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891895-p4bzz" event={"ID":"0d62a50d-6bb5-4200-bc69-70deaa07d3d3","Type":"ContainerDied","Data":"2947d2e6837f3d692d84db1de6d3ae802c4859384b782fb207b1c00219b0a7d4"} Nov 01 06:15:19.400334 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.400305 5040 scope.go:122] "RemoveContainer" containerID="90ca8867c251d0aa5ae6ca73a2a5c31bb4ef06322a0ca9d587746ee82e21ef65" Nov 01 06:15:19.403665 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.403544 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e004500a-tmq5k" Nov 01 06:15:19.413209 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.413180 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3578ce-29891895-5vnj9"] Nov 01 06:15:19.417570 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.417548 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3578ce-29891895-5vnj9"] Nov 01 06:15:19.515749 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.515717 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25021d94-5042-4ee0-b69f-12ceb0564036" path="/var/lib/kubelet/pods/25021d94-5042-4ee0-b69f-12ceb0564036/volumes" Nov 01 06:15:19.516282 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.516265 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b1f2aca-3d31-4c8d-b83a-ca20a360e75e" path="/var/lib/kubelet/pods/3b1f2aca-3d31-4c8d-b83a-ca20a360e75e/volumes" Nov 01 06:15:19.516802 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.516786 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b7f1411-aaf9-4d3b-91c0-40f875416c57" path="/var/lib/kubelet/pods/7b7f1411-aaf9-4d3b-91c0-40f875416c57/volumes" Nov 01 06:15:19.517737 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.517718 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec119deb-0b81-447c-af06-9464177b7a01" path="/var/lib/kubelet/pods/ec119deb-0b81-447c-af06-9464177b7a01/volumes" Nov 01 06:15:19.532433 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.532405 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891895-p4bzz"] Nov 01 06:15:19.536184 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.536164 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-w8t88" Nov 01 06:15:19.537913 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.537890 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891895-p4bzz"] Nov 01 06:15:19.649208 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.649173 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78e5172d-422f-4082-9f96-15f1ab8b49eb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78e5172d-422f-4082-9f96-15f1ab8b49eb-maas-api-service-ca\") pod \"78e5172d-422f-4082-9f96-15f1ab8b49eb\" (UID: \"78e5172d-422f-4082-9f96-15f1ab8b49eb\") " Nov 01 06:15:19.649373 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.649224 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78e5172d-422f-4082-9f96-15f1ab8b49eb-kube-api-access-jlcpt\" (UniqueName: \"kubernetes.io/projected/78e5172d-422f-4082-9f96-15f1ab8b49eb-kube-api-access-jlcpt\") pod \"78e5172d-422f-4082-9f96-15f1ab8b49eb\" (UID: \"78e5172d-422f-4082-9f96-15f1ab8b49eb\") " Nov 01 06:15:19.649573 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.649548 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78e5172d-422f-4082-9f96-15f1ab8b49eb-maas-api-service-ca" pod "78e5172d-422f-4082-9f96-15f1ab8b49eb" (UID: "78e5172d-422f-4082-9f96-15f1ab8b49eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:15:19.651604 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.651570 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78e5172d-422f-4082-9f96-15f1ab8b49eb-kube-api-access-jlcpt" pod "78e5172d-422f-4082-9f96-15f1ab8b49eb" (UID: "78e5172d-422f-4082-9f96-15f1ab8b49eb"). InnerVolumeSpecName "kube-api-access-jlcpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:15:19.750796 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.750758 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78e5172d-422f-4082-9f96-15f1ab8b49eb-maas-api-service-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:19.750796 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.750789 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlcpt\" (UniqueName: \"kubernetes.io/projected/78e5172d-422f-4082-9f96-15f1ab8b49eb-kube-api-access-jlcpt\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:19.763046 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.763013 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891895-h46r4"] Nov 01 06:15:19.766907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.766842 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891895-h46r4"] Nov 01 06:15:19.793756 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:19.793731 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e004500a-tmq5k"] Nov 01 06:15:19.795205 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:15:19.795176 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89cda59d_10d7_434f_890e_6e5c5b1caf45.slice/crio-9c42f2f480fbd08dbb902f83e9f15f6f7487a606a847b19d0527724c676152bd WatchSource:0}: Error finding container 9c42f2f480fbd08dbb902f83e9f15f6f7487a606a847b19d0527724c676152bd: Status 404 returned error can't find the container with id 9c42f2f480fbd08dbb902f83e9f15f6f7487a606a847b19d0527724c676152bd Nov 01 06:15:20.387885 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:20.387795 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-w8t88" Nov 01 06:15:20.387885 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:20.387835 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-w8t88" event={"ID":"78e5172d-422f-4082-9f96-15f1ab8b49eb","Type":"ContainerDied","Data":"f393f823c64a5804cdb28ae66c0a0b6eb007440499076463f8c8db8b76a3dc6c"} Nov 01 06:15:20.388441 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:20.387901 5040 scope.go:122] "RemoveContainer" containerID="1cde6b7767c721a564bf27b419d18689d9da970ff302d5b05563b2de4e662438" Nov 01 06:15:20.390244 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:20.390202 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e004500a-tmq5k" event={"ID":"89cda59d-10d7-434f-890e-6e5c5b1caf45","Type":"ContainerStarted","Data":"78410d69807f3ef368d852ededfaf9e9b2938416c3ec90ab897d3a13448bc7ae"} Nov 01 06:15:20.390371 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:20.390273 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e004500a-tmq5k" event={"ID":"89cda59d-10d7-434f-890e-6e5c5b1caf45","Type":"ContainerStarted","Data":"9c42f2f480fbd08dbb902f83e9f15f6f7487a606a847b19d0527724c676152bd"} Nov 01 06:15:20.557172 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:20.557123 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e004500a-tmq5k" podStartSLOduration=1.5571093230000002 podStartE2EDuration="1.557109323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:15:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:15:20.556498746 +0000 UTC m=+1799.211615915" watchObservedRunningTime="2026-11-01 06:15:20.557109323 +0000 UTC m=+1799.212226491" Nov 01 06:15:20.671971 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:20.671942 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891895-w8t88"] Nov 01 06:15:20.676004 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:20.675976 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891895-w8t88"] Nov 01 06:15:21.399869 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:21.399836 5040 generic.go:356] "Generic (PLEG): container finished" podID="89cda59d-10d7-434f-890e-6e5c5b1caf45" containerID="78410d69807f3ef368d852ededfaf9e9b2938416c3ec90ab897d3a13448bc7ae" exitCode=0 Nov 01 06:15:21.400335 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:21.399932 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e004500a-tmq5k" event={"ID":"89cda59d-10d7-434f-890e-6e5c5b1caf45","Type":"ContainerDied","Data":"78410d69807f3ef368d852ededfaf9e9b2938416c3ec90ab897d3a13448bc7ae"} Nov 01 06:15:21.516583 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:21.516540 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d62a50d-6bb5-4200-bc69-70deaa07d3d3" path="/var/lib/kubelet/pods/0d62a50d-6bb5-4200-bc69-70deaa07d3d3/volumes" Nov 01 06:15:21.517138 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:21.517113 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78e5172d-422f-4082-9f96-15f1ab8b49eb" path="/var/lib/kubelet/pods/78e5172d-422f-4082-9f96-15f1ab8b49eb/volumes" Nov 01 06:15:21.517698 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:21.517676 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6610260-baea-4921-9a3a-756fbf18eb3e" path="/var/lib/kubelet/pods/c6610260-baea-4921-9a3a-756fbf18eb3e/volumes" Nov 01 06:15:21.784154 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:21.784120 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 06:15:21.791446 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:21.791420 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 06:15:22.499760 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:22.499728 5040 scope.go:122] "RemoveContainer" containerID="fa450ce4c63c03a45a7e2ab4f0f7c173a2c3a4bacf483052e8aff329ba46e1a7" Nov 01 06:15:22.515595 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:22.515552 5040 scope.go:122] "RemoveContainer" containerID="7ef8bf1798e27eacf827ebe3651ea967260c4aae4b14bf2ed92d2fdd81f98158" Nov 01 06:15:22.568375 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:22.568355 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e004500a-tmq5k" Nov 01 06:15:22.683728 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:22.683643 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89cda59d-10d7-434f-890e-6e5c5b1caf45-kube-api-access-8jmvx\" (UniqueName: \"kubernetes.io/projected/89cda59d-10d7-434f-890e-6e5c5b1caf45-kube-api-access-8jmvx\") pod \"89cda59d-10d7-434f-890e-6e5c5b1caf45\" (UID: \"89cda59d-10d7-434f-890e-6e5c5b1caf45\") " Nov 01 06:15:22.683728 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:22.683695 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89cda59d-10d7-434f-890e-6e5c5b1caf45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89cda59d-10d7-434f-890e-6e5c5b1caf45-maas-api-openshift-ca\") pod \"89cda59d-10d7-434f-890e-6e5c5b1caf45\" (UID: \"89cda59d-10d7-434f-890e-6e5c5b1caf45\") " Nov 01 06:15:22.683919 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:22.683778 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89cda59d-10d7-434f-890e-6e5c5b1caf45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89cda59d-10d7-434f-890e-6e5c5b1caf45-maas-api-xks-ca\") pod \"89cda59d-10d7-434f-890e-6e5c5b1caf45\" (UID: \"89cda59d-10d7-434f-890e-6e5c5b1caf45\") " Nov 01 06:15:22.684069 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:22.684042 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89cda59d-10d7-434f-890e-6e5c5b1caf45-maas-api-openshift-ca" pod "89cda59d-10d7-434f-890e-6e5c5b1caf45" (UID: "89cda59d-10d7-434f-890e-6e5c5b1caf45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:15:22.684148 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:22.684126 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89cda59d-10d7-434f-890e-6e5c5b1caf45-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:22.686428 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:22.686320 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89cda59d-10d7-434f-890e-6e5c5b1caf45-kube-api-access-8jmvx" pod "89cda59d-10d7-434f-890e-6e5c5b1caf45" (UID: "89cda59d-10d7-434f-890e-6e5c5b1caf45"). InnerVolumeSpecName "kube-api-access-8jmvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:15:22.686428 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:22.686339 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89cda59d-10d7-434f-890e-6e5c5b1caf45-maas-api-xks-ca" pod "89cda59d-10d7-434f-890e-6e5c5b1caf45" (UID: "89cda59d-10d7-434f-890e-6e5c5b1caf45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:15:22.785000 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:22.784961 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jmvx\" (UniqueName: \"kubernetes.io/projected/89cda59d-10d7-434f-890e-6e5c5b1caf45-kube-api-access-8jmvx\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:22.785000 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:22.784993 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89cda59d-10d7-434f-890e-6e5c5b1caf45-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:23.415283 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:23.415254 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e004500a-tmq5k" Nov 01 06:15:23.415469 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:23.415295 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e004500a-tmq5k" event={"ID":"89cda59d-10d7-434f-890e-6e5c5b1caf45","Type":"ContainerDied","Data":"9c42f2f480fbd08dbb902f83e9f15f6f7487a606a847b19d0527724c676152bd"} Nov 01 06:15:23.415469 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:23.415341 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c42f2f480fbd08dbb902f83e9f15f6f7487a606a847b19d0527724c676152bd" Nov 01 06:15:24.399387 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:24.399348 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl"] Nov 01 06:15:24.400013 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:24.399672 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" podUID="8f892105-215a-45be-88eb-ac5f03c50c6f" containerName="istio-proxy" containerID="cri-o://8d9b161c59493586ca1b206ecd47fb4d23a6ab568bbf2e36e78720af3225727e" gracePeriod=30 Nov 01 06:15:29.660440 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.660414 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" Nov 01 06:15:29.759926 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.759899 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f892105-215a-45be-88eb-ac5f03c50c6f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f892105-215a-45be-88eb-ac5f03c50c6f-istio-data\") pod \"8f892105-215a-45be-88eb-ac5f03c50c6f\" (UID: \"8f892105-215a-45be-88eb-ac5f03c50c6f\") " Nov 01 06:15:29.760067 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.759942 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f892105-215a-45be-88eb-ac5f03c50c6f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f892105-215a-45be-88eb-ac5f03c50c6f-workload-socket\") pod \"8f892105-215a-45be-88eb-ac5f03c50c6f\" (UID: \"8f892105-215a-45be-88eb-ac5f03c50c6f\") " Nov 01 06:15:29.760067 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.759991 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f892105-215a-45be-88eb-ac5f03c50c6f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f892105-215a-45be-88eb-ac5f03c50c6f-istio-podinfo\") pod \"8f892105-215a-45be-88eb-ac5f03c50c6f\" (UID: \"8f892105-215a-45be-88eb-ac5f03c50c6f\") " Nov 01 06:15:29.760067 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.760030 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f892105-215a-45be-88eb-ac5f03c50c6f-kube-api-access-b2vqb\" (UniqueName: \"kubernetes.io/projected/8f892105-215a-45be-88eb-ac5f03c50c6f-kube-api-access-b2vqb\") pod \"8f892105-215a-45be-88eb-ac5f03c50c6f\" (UID: \"8f892105-215a-45be-88eb-ac5f03c50c6f\") " Nov 01 06:15:29.760206 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.760064 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f892105-215a-45be-88eb-ac5f03c50c6f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f892105-215a-45be-88eb-ac5f03c50c6f-workload-certs\") pod \"8f892105-215a-45be-88eb-ac5f03c50c6f\" (UID: \"8f892105-215a-45be-88eb-ac5f03c50c6f\") " Nov 01 06:15:29.760206 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.760128 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f892105-215a-45be-88eb-ac5f03c50c6f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f892105-215a-45be-88eb-ac5f03c50c6f-istiod-ca-cert\") pod \"8f892105-215a-45be-88eb-ac5f03c50c6f\" (UID: \"8f892105-215a-45be-88eb-ac5f03c50c6f\") " Nov 01 06:15:29.760303 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.760241 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f892105-215a-45be-88eb-ac5f03c50c6f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f892105-215a-45be-88eb-ac5f03c50c6f-istio-envoy\") pod \"8f892105-215a-45be-88eb-ac5f03c50c6f\" (UID: \"8f892105-215a-45be-88eb-ac5f03c50c6f\") " Nov 01 06:15:29.760303 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.760217 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f892105-215a-45be-88eb-ac5f03c50c6f-istio-data" pod "8f892105-215a-45be-88eb-ac5f03c50c6f" (UID: "8f892105-215a-45be-88eb-ac5f03c50c6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:15:29.760407 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.760295 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f892105-215a-45be-88eb-ac5f03c50c6f-workload-socket" pod "8f892105-215a-45be-88eb-ac5f03c50c6f" (UID: "8f892105-215a-45be-88eb-ac5f03c50c6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:15:29.760407 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.760332 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f892105-215a-45be-88eb-ac5f03c50c6f-istio-token\" (UniqueName: \"kubernetes.io/projected/8f892105-215a-45be-88eb-ac5f03c50c6f-istio-token\") pod \"8f892105-215a-45be-88eb-ac5f03c50c6f\" (UID: \"8f892105-215a-45be-88eb-ac5f03c50c6f\") " Nov 01 06:15:29.760407 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.760371 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f892105-215a-45be-88eb-ac5f03c50c6f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f892105-215a-45be-88eb-ac5f03c50c6f-credential-socket\") pod \"8f892105-215a-45be-88eb-ac5f03c50c6f\" (UID: \"8f892105-215a-45be-88eb-ac5f03c50c6f\") " Nov 01 06:15:29.760584 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.760525 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f892105-215a-45be-88eb-ac5f03c50c6f-workload-certs" pod "8f892105-215a-45be-88eb-ac5f03c50c6f" (UID: "8f892105-215a-45be-88eb-ac5f03c50c6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:15:29.760792 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.760737 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f892105-215a-45be-88eb-ac5f03c50c6f-credential-socket" pod "8f892105-215a-45be-88eb-ac5f03c50c6f" (UID: "8f892105-215a-45be-88eb-ac5f03c50c6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:15:29.760792 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.760650 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f892105-215a-45be-88eb-ac5f03c50c6f-istiod-ca-cert" pod "8f892105-215a-45be-88eb-ac5f03c50c6f" (UID: "8f892105-215a-45be-88eb-ac5f03c50c6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:15:29.760965 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.760810 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f892105-215a-45be-88eb-ac5f03c50c6f-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:29.760965 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.760832 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f892105-215a-45be-88eb-ac5f03c50c6f-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:29.760965 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.760849 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f892105-215a-45be-88eb-ac5f03c50c6f-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:29.762445 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.762417 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f892105-215a-45be-88eb-ac5f03c50c6f-istio-envoy" pod "8f892105-215a-45be-88eb-ac5f03c50c6f" (UID: "8f892105-215a-45be-88eb-ac5f03c50c6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:15:29.762552 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.762441 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f892105-215a-45be-88eb-ac5f03c50c6f-istio-token" pod "8f892105-215a-45be-88eb-ac5f03c50c6f" (UID: "8f892105-215a-45be-88eb-ac5f03c50c6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:15:29.762743 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.762723 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f892105-215a-45be-88eb-ac5f03c50c6f-kube-api-access-b2vqb" pod "8f892105-215a-45be-88eb-ac5f03c50c6f" (UID: "8f892105-215a-45be-88eb-ac5f03c50c6f"). InnerVolumeSpecName "kube-api-access-b2vqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:15:29.762809 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.762741 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f892105-215a-45be-88eb-ac5f03c50c6f-istio-podinfo" pod "8f892105-215a-45be-88eb-ac5f03c50c6f" (UID: "8f892105-215a-45be-88eb-ac5f03c50c6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:15:29.835590 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.835549 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77d25194-klk5n"] Nov 01 06:15:29.836024 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.836006 5040 cpu_manager.go:414] "containerMap: removing container" podUID="7b7f1411-aaf9-4d3b-91c0-40f875416c57" containerName="cleanup" Nov 01 06:15:29.836129 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.836026 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b7f1411-aaf9-4d3b-91c0-40f875416c57" containerName="cleanup" Nov 01 06:15:29.836129 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.836047 5040 cpu_manager.go:414] "containerMap: removing container" podUID="25021d94-5042-4ee0-b69f-12ceb0564036" containerName="cleanup" Nov 01 06:15:29.836129 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.836055 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25021d94-5042-4ee0-b69f-12ceb0564036" containerName="cleanup" Nov 01 06:15:29.836129 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.836067 5040 cpu_manager.go:414] "containerMap: removing container" podUID="89cda59d-10d7-434f-890e-6e5c5b1caf45" containerName="revoke-keys" Nov 01 06:15:29.836129 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.836074 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89cda59d-10d7-434f-890e-6e5c5b1caf45" containerName="revoke-keys" Nov 01 06:15:29.836129 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.836087 5040 cpu_manager.go:414] "containerMap: removing container" podUID="3b1f2aca-3d31-4c8d-b83a-ca20a360e75e" containerName="cleanup" Nov 01 06:15:29.836129 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.836094 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b1f2aca-3d31-4c8d-b83a-ca20a360e75e" containerName="cleanup" Nov 01 06:15:29.836520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.836137 5040 cpu_manager.go:414] "containerMap: removing container" podUID="78e5172d-422f-4082-9f96-15f1ab8b49eb" containerName="cleanup" Nov 01 06:15:29.836520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.836144 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78e5172d-422f-4082-9f96-15f1ab8b49eb" containerName="cleanup" Nov 01 06:15:29.836520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.836158 5040 cpu_manager.go:414] "containerMap: removing container" podUID="8f892105-215a-45be-88eb-ac5f03c50c6f" containerName="istio-proxy" Nov 01 06:15:29.836520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.836166 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f892105-215a-45be-88eb-ac5f03c50c6f" containerName="istio-proxy" Nov 01 06:15:29.836520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.836180 5040 cpu_manager.go:414] "containerMap: removing container" podUID="c6610260-baea-4921-9a3a-756fbf18eb3e" containerName="cleanup" Nov 01 06:15:29.836520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.836187 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6610260-baea-4921-9a3a-756fbf18eb3e" containerName="cleanup" Nov 01 06:15:29.836520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.836202 5040 cpu_manager.go:414] "containerMap: removing container" podUID="78e5172d-422f-4082-9f96-15f1ab8b49eb" containerName="cleanup" Nov 01 06:15:29.836520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.836210 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78e5172d-422f-4082-9f96-15f1ab8b49eb" containerName="cleanup" Nov 01 06:15:29.836520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.836221 5040 cpu_manager.go:414] "containerMap: removing container" podUID="78e5172d-422f-4082-9f96-15f1ab8b49eb" containerName="cleanup" Nov 01 06:15:29.836520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.836228 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78e5172d-422f-4082-9f96-15f1ab8b49eb" containerName="cleanup" Nov 01 06:15:29.836520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.836248 5040 cpu_manager.go:414] "containerMap: removing container" podUID="0d62a50d-6bb5-4200-bc69-70deaa07d3d3" containerName="cleanup" Nov 01 06:15:29.836520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.836255 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d62a50d-6bb5-4200-bc69-70deaa07d3d3" containerName="cleanup" Nov 01 06:15:29.836520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.836375 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="89cda59d-10d7-434f-890e-6e5c5b1caf45" containerName="revoke-keys" Nov 01 06:15:29.836520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.836390 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec119deb-0b81-447c-af06-9464177b7a01" containerName="cleanup" Nov 01 06:15:29.836520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.836400 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b7f1411-aaf9-4d3b-91c0-40f875416c57" containerName="cleanup" Nov 01 06:15:29.836520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.836410 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d62a50d-6bb5-4200-bc69-70deaa07d3d3" containerName="cleanup" Nov 01 06:15:29.836520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.836424 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="78e5172d-422f-4082-9f96-15f1ab8b49eb" containerName="cleanup" Nov 01 06:15:29.836520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.836434 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f892105-215a-45be-88eb-ac5f03c50c6f" containerName="istio-proxy" Nov 01 06:15:29.836520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.836444 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="78e5172d-422f-4082-9f96-15f1ab8b49eb" containerName="cleanup" Nov 01 06:15:29.836520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.836453 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="78e5172d-422f-4082-9f96-15f1ab8b49eb" containerName="cleanup" Nov 01 06:15:29.836520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.836462 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6610260-baea-4921-9a3a-756fbf18eb3e" containerName="cleanup" Nov 01 06:15:29.840292 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.840270 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d25194-klk5n" Nov 01 06:15:29.861727 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.861705 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f892105-215a-45be-88eb-ac5f03c50c6f-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:29.861727 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.861727 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f892105-215a-45be-88eb-ac5f03c50c6f-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:29.861862 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.861740 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f892105-215a-45be-88eb-ac5f03c50c6f-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:29.861862 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.861755 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2vqb\" (UniqueName: \"kubernetes.io/projected/8f892105-215a-45be-88eb-ac5f03c50c6f-kube-api-access-b2vqb\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:29.861862 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.861764 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f892105-215a-45be-88eb-ac5f03c50c6f-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:29.861862 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.861773 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f892105-215a-45be-88eb-ac5f03c50c6f-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:29.893962 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.893897 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:15:29.894379 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.894351 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:15:29.894782 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.894572 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:15:29.894782 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.894648 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:15:29.897598 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.897574 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77d25194-klk5n"] Nov 01 06:15:29.962823 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.962786 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a810b4d-d7e5-4d1b-8299-71fae1652240-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77d25194-klk5n\" (UID: \"7a810b4d-d7e5-4d1b-8299-71fae1652240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d25194-klk5n" Nov 01 06:15:29.962993 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.962831 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a810b4d-d7e5-4d1b-8299-71fae1652240-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77d25194-klk5n\" (UID: \"7a810b4d-d7e5-4d1b-8299-71fae1652240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d25194-klk5n" Nov 01 06:15:29.962993 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:29.962862 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cxfn\" (UniqueName: \"kubernetes.io/projected/7a810b4d-d7e5-4d1b-8299-71fae1652240-kube-api-access-4cxfn\") pod \"maas-api-revoke-sub-77d25194-klk5n\" (UID: \"7a810b4d-d7e5-4d1b-8299-71fae1652240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d25194-klk5n" Nov 01 06:15:30.063715 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.063639 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a810b4d-d7e5-4d1b-8299-71fae1652240-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77d25194-klk5n\" (UID: \"7a810b4d-d7e5-4d1b-8299-71fae1652240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d25194-klk5n" Nov 01 06:15:30.063715 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.063681 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a810b4d-d7e5-4d1b-8299-71fae1652240-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77d25194-klk5n\" (UID: \"7a810b4d-d7e5-4d1b-8299-71fae1652240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d25194-klk5n" Nov 01 06:15:30.063715 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.063708 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cxfn\" (UniqueName: \"kubernetes.io/projected/7a810b4d-d7e5-4d1b-8299-71fae1652240-kube-api-access-4cxfn\") pod \"maas-api-revoke-sub-77d25194-klk5n\" (UID: \"7a810b4d-d7e5-4d1b-8299-71fae1652240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d25194-klk5n" Nov 01 06:15:30.064313 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.064291 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a810b4d-d7e5-4d1b-8299-71fae1652240-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77d25194-klk5n\" (UID: \"7a810b4d-d7e5-4d1b-8299-71fae1652240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d25194-klk5n" Nov 01 06:15:30.066034 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.066015 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a810b4d-d7e5-4d1b-8299-71fae1652240-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77d25194-klk5n\" (UID: \"7a810b4d-d7e5-4d1b-8299-71fae1652240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d25194-klk5n" Nov 01 06:15:30.125748 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.125712 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cxfn\" (UniqueName: \"kubernetes.io/projected/7a810b4d-d7e5-4d1b-8299-71fae1652240-kube-api-access-4cxfn\") pod \"maas-api-revoke-sub-77d25194-klk5n\" (UID: \"7a810b4d-d7e5-4d1b-8299-71fae1652240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d25194-klk5n" Nov 01 06:15:30.153457 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.153427 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d25194-klk5n" Nov 01 06:15:30.163290 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.163260 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm"] Nov 01 06:15:30.167377 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.167358 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm" Nov 01 06:15:30.223925 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.221514 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7d25db\"/\"agc-cred-b2178f695dad\"" Nov 01 06:15:30.223925 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.221657 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7d25db\"/\"payload-processing-external-model-plugins-e2e-extmod-b-7d25db\"" Nov 01 06:15:30.223925 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.221739 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7d25db\"/\"routing-overlay\"" Nov 01 06:15:30.223925 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.221849 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7d25db\"/\"openshift-service-ca.crt\"" Nov 01 06:15:30.223925 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.221974 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7d25db\"/\"payload-processing-external-model-e2e--a835acb2-dockercfg-kqgqq\"" Nov 01 06:15:30.223925 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.222048 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7d25db\"/\"agc-cred-bdcae79313a2\"" Nov 01 06:15:30.223925 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.222144 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7d25db\"/\"kube-root-ca.crt\"" Nov 01 06:15:30.224399 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.223987 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm"] Nov 01 06:15:30.266093 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.266057 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9decea03-910e-4838-b4f0-b6065bf7cf8b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm\" (UID: \"9decea03-910e-4838-b4f0-b6065bf7cf8b\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm" Nov 01 06:15:30.266268 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.266248 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9decea03-910e-4838-b4f0-b6065bf7cf8b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm\" (UID: \"9decea03-910e-4838-b4f0-b6065bf7cf8b\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm" Nov 01 06:15:30.266327 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.266284 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9decea03-910e-4838-b4f0-b6065bf7cf8b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm\" (UID: \"9decea03-910e-4838-b4f0-b6065bf7cf8b\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm" Nov 01 06:15:30.266450 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.266425 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vzcv\" (UniqueName: \"kubernetes.io/projected/9decea03-910e-4838-b4f0-b6065bf7cf8b-kube-api-access-2vzcv\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm\" (UID: \"9decea03-910e-4838-b4f0-b6065bf7cf8b\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm" Nov 01 06:15:30.367801 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.367717 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9decea03-910e-4838-b4f0-b6065bf7cf8b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm\" (UID: \"9decea03-910e-4838-b4f0-b6065bf7cf8b\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm" Nov 01 06:15:30.367801 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.367755 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9decea03-910e-4838-b4f0-b6065bf7cf8b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm\" (UID: \"9decea03-910e-4838-b4f0-b6065bf7cf8b\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm" Nov 01 06:15:30.367801 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.367784 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vzcv\" (UniqueName: \"kubernetes.io/projected/9decea03-910e-4838-b4f0-b6065bf7cf8b-kube-api-access-2vzcv\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm\" (UID: \"9decea03-910e-4838-b4f0-b6065bf7cf8b\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm" Nov 01 06:15:30.368044 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.367830 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9decea03-910e-4838-b4f0-b6065bf7cf8b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm\" (UID: \"9decea03-910e-4838-b4f0-b6065bf7cf8b\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm" Nov 01 06:15:30.368427 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.368405 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9decea03-910e-4838-b4f0-b6065bf7cf8b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm\" (UID: \"9decea03-910e-4838-b4f0-b6065bf7cf8b\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm" Nov 01 06:15:30.368537 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.368467 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9decea03-910e-4838-b4f0-b6065bf7cf8b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm\" (UID: \"9decea03-910e-4838-b4f0-b6065bf7cf8b\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm" Nov 01 06:15:30.370350 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.370329 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9decea03-910e-4838-b4f0-b6065bf7cf8b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm\" (UID: \"9decea03-910e-4838-b4f0-b6065bf7cf8b\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm" Nov 01 06:15:30.429758 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.429728 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vzcv\" (UniqueName: \"kubernetes.io/projected/9decea03-910e-4838-b4f0-b6065bf7cf8b-kube-api-access-2vzcv\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm\" (UID: \"9decea03-910e-4838-b4f0-b6065bf7cf8b\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm" Nov 01 06:15:30.460247 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.460211 5040 generic.go:356] "Generic (PLEG): container finished" podID="8f892105-215a-45be-88eb-ac5f03c50c6f" containerID="8d9b161c59493586ca1b206ecd47fb4d23a6ab568bbf2e36e78720af3225727e" exitCode=0 Nov 01 06:15:30.460388 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.460263 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" Nov 01 06:15:30.460388 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.460297 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" event={"ID":"8f892105-215a-45be-88eb-ac5f03c50c6f","Type":"ContainerDied","Data":"8d9b161c59493586ca1b206ecd47fb4d23a6ab568bbf2e36e78720af3225727e"} Nov 01 06:15:30.460388 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.460353 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl" event={"ID":"8f892105-215a-45be-88eb-ac5f03c50c6f","Type":"ContainerDied","Data":"afb2c597c6993658ed70cf93be5e5f9e3b0c91d90667e8b5d392caf40f73c340"} Nov 01 06:15:30.460388 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.460376 5040 scope.go:122] "RemoveContainer" containerID="8d9b161c59493586ca1b206ecd47fb4d23a6ab568bbf2e36e78720af3225727e" Nov 01 06:15:30.471125 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.471108 5040 scope.go:122] "RemoveContainer" containerID="8d9b161c59493586ca1b206ecd47fb4d23a6ab568bbf2e36e78720af3225727e" Nov 01 06:15:30.471409 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:15:30.471392 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d9b161c59493586ca1b206ecd47fb4d23a6ab568bbf2e36e78720af3225727e\": container with ID starting with 8d9b161c59493586ca1b206ecd47fb4d23a6ab568bbf2e36e78720af3225727e not found: ID does not exist" containerID="8d9b161c59493586ca1b206ecd47fb4d23a6ab568bbf2e36e78720af3225727e" Nov 01 06:15:30.471457 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.471420 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d9b161c59493586ca1b206ecd47fb4d23a6ab568bbf2e36e78720af3225727e"} err="failed to get container status \"8d9b161c59493586ca1b206ecd47fb4d23a6ab568bbf2e36e78720af3225727e\": rpc error: code = NotFound desc = could not find container \"8d9b161c59493586ca1b206ecd47fb4d23a6ab568bbf2e36e78720af3225727e\": container with ID starting with 8d9b161c59493586ca1b206ecd47fb4d23a6ab568bbf2e36e78720af3225727e not found: ID does not exist" Nov 01 06:15:30.481123 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.481102 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm" Nov 01 06:15:30.530364 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.530333 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77d25194-klk5n"] Nov 01 06:15:30.531210 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:15:30.531185 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a810b4d_d7e5_4d1b_8299_71fae1652240.slice/crio-3c5117af985ce4953ef8d6bc559666afd0a6d05d24b8dce7999c4708e0456714 WatchSource:0}: Error finding container 3c5117af985ce4953ef8d6bc559666afd0a6d05d24b8dce7999c4708e0456714: Status 404 returned error can't find the container with id 3c5117af985ce4953ef8d6bc559666afd0a6d05d24b8dce7999c4708e0456714 Nov 01 06:15:30.628858 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.628782 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl"] Nov 01 06:15:30.634370 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.634345 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kw9tl"] Nov 01 06:15:30.863817 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:30.863791 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm"] Nov 01 06:15:30.864834 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:15:30.864812 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9decea03_910e_4838_b4f0_b6065bf7cf8b.slice/crio-efe7a1b349c12a4dd488924e5d2786697e66dac567c12cae363f31798be9daa6 WatchSource:0}: Error finding container efe7a1b349c12a4dd488924e5d2786697e66dac567c12cae363f31798be9daa6: Status 404 returned error can't find the container with id efe7a1b349c12a4dd488924e5d2786697e66dac567c12cae363f31798be9daa6 Nov 01 06:15:31.468795 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:31.468762 5040 generic.go:356] "Generic (PLEG): container finished" podID="7a810b4d-d7e5-4d1b-8299-71fae1652240" containerID="1ea9a5207294cefd558fa2313f9e14a41261002dbc9488e8ee85fd71f9ff7a20" exitCode=0 Nov 01 06:15:31.468985 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:31.468854 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d25194-klk5n" event={"ID":"7a810b4d-d7e5-4d1b-8299-71fae1652240","Type":"ContainerDied","Data":"1ea9a5207294cefd558fa2313f9e14a41261002dbc9488e8ee85fd71f9ff7a20"} Nov 01 06:15:31.468985 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:31.468916 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d25194-klk5n" event={"ID":"7a810b4d-d7e5-4d1b-8299-71fae1652240","Type":"ContainerStarted","Data":"3c5117af985ce4953ef8d6bc559666afd0a6d05d24b8dce7999c4708e0456714"} Nov 01 06:15:31.470354 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:31.470331 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm" event={"ID":"9decea03-910e-4838-b4f0-b6065bf7cf8b","Type":"ContainerStarted","Data":"527532ce574fabca966ed05cf4410ae3c90456b5afca18773e7e1fcc65329f0e"} Nov 01 06:15:31.470467 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:31.470373 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm" event={"ID":"9decea03-910e-4838-b4f0-b6065bf7cf8b","Type":"ContainerStarted","Data":"efe7a1b349c12a4dd488924e5d2786697e66dac567c12cae363f31798be9daa6"} Nov 01 06:15:31.470467 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:31.470401 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm" Nov 01 06:15:31.514331 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:31.514295 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f892105-215a-45be-88eb-ac5f03c50c6f" path="/var/lib/kubelet/pods/8f892105-215a-45be-88eb-ac5f03c50c6f/volumes" Nov 01 06:15:31.705095 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:31.704990 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm" podStartSLOduration=1.7049764779999999 podStartE2EDuration="1.704976478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:15:31.702270686 +0000 UTC m=+1810.357387853" watchObservedRunningTime="2026-11-01 06:15:31.704976478 +0000 UTC m=+1810.360093644" Nov 01 06:15:32.630778 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:32.630747 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d25194-klk5n" Nov 01 06:15:32.740771 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:32.740671 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h"] Nov 01 06:15:32.741247 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:32.741227 5040 cpu_manager.go:414] "containerMap: removing container" podUID="7a810b4d-d7e5-4d1b-8299-71fae1652240" containerName="revoke-keys" Nov 01 06:15:32.741338 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:32.741249 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a810b4d-d7e5-4d1b-8299-71fae1652240" containerName="revoke-keys" Nov 01 06:15:32.741809 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:32.741372 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a810b4d-d7e5-4d1b-8299-71fae1652240" containerName="revoke-keys" Nov 01 06:15:32.744718 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:32.744697 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h" Nov 01 06:15:32.758912 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:32.758889 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h"] Nov 01 06:15:32.793098 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:32.793068 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a810b4d-d7e5-4d1b-8299-71fae1652240-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a810b4d-d7e5-4d1b-8299-71fae1652240-maas-api-openshift-ca\") pod \"7a810b4d-d7e5-4d1b-8299-71fae1652240\" (UID: \"7a810b4d-d7e5-4d1b-8299-71fae1652240\") " Nov 01 06:15:32.793281 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:32.793133 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a810b4d-d7e5-4d1b-8299-71fae1652240-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a810b4d-d7e5-4d1b-8299-71fae1652240-maas-api-xks-ca\") pod \"7a810b4d-d7e5-4d1b-8299-71fae1652240\" (UID: \"7a810b4d-d7e5-4d1b-8299-71fae1652240\") " Nov 01 06:15:32.793281 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:32.793162 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a810b4d-d7e5-4d1b-8299-71fae1652240-kube-api-access-4cxfn\" (UniqueName: \"kubernetes.io/projected/7a810b4d-d7e5-4d1b-8299-71fae1652240-kube-api-access-4cxfn\") pod \"7a810b4d-d7e5-4d1b-8299-71fae1652240\" (UID: \"7a810b4d-d7e5-4d1b-8299-71fae1652240\") " Nov 01 06:15:32.793538 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:32.793511 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a810b4d-d7e5-4d1b-8299-71fae1652240-maas-api-openshift-ca" pod "7a810b4d-d7e5-4d1b-8299-71fae1652240" (UID: "7a810b4d-d7e5-4d1b-8299-71fae1652240"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:15:32.793616 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:32.793583 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a810b4d-d7e5-4d1b-8299-71fae1652240-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:32.795333 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:32.795303 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a810b4d-d7e5-4d1b-8299-71fae1652240-maas-api-xks-ca" pod "7a810b4d-d7e5-4d1b-8299-71fae1652240" (UID: "7a810b4d-d7e5-4d1b-8299-71fae1652240"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:15:32.795416 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:32.795390 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a810b4d-d7e5-4d1b-8299-71fae1652240-kube-api-access-4cxfn" pod "7a810b4d-d7e5-4d1b-8299-71fae1652240" (UID: "7a810b4d-d7e5-4d1b-8299-71fae1652240"). InnerVolumeSpecName "kube-api-access-4cxfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:15:32.799515 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:32.799463 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"payload-processing-external-model-plugins-e2e-extmod-a-8f45a2\"" Nov 01 06:15:32.799624 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:32.799545 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"agc-cred-c481d4f49e7f\"" Nov 01 06:15:32.799917 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:32.799898 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"openshift-service-ca.crt\"" Nov 01 06:15:32.799991 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:32.799898 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"agc-cred-d6d3484085b2\"" Nov 01 06:15:32.799991 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:32.799938 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"payload-processing-external-model-e2e--cb1d2723-dockercfg-smk8q\"" Nov 01 06:15:32.800177 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:32.800163 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"routing-overlay\"" Nov 01 06:15:32.800212 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:32.800176 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"kube-root-ca.crt\"" Nov 01 06:15:32.894961 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:32.894920 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6mnx\" (UniqueName: \"kubernetes.io/projected/f6f444d8-62e1-4f3f-b7dd-9d31399d7534-kube-api-access-h6mnx\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h\" (UID: \"f6f444d8-62e1-4f3f-b7dd-9d31399d7534\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h" Nov 01 06:15:32.895125 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:32.894984 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f6f444d8-62e1-4f3f-b7dd-9d31399d7534-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h\" (UID: \"f6f444d8-62e1-4f3f-b7dd-9d31399d7534\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h" Nov 01 06:15:32.895194 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:32.895133 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f6f444d8-62e1-4f3f-b7dd-9d31399d7534-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h\" (UID: \"f6f444d8-62e1-4f3f-b7dd-9d31399d7534\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h" Nov 01 06:15:32.895194 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:32.895174 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6f444d8-62e1-4f3f-b7dd-9d31399d7534-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h\" (UID: \"f6f444d8-62e1-4f3f-b7dd-9d31399d7534\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h" Nov 01 06:15:32.895273 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:32.895244 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a810b4d-d7e5-4d1b-8299-71fae1652240-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:32.895273 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:32.895259 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cxfn\" (UniqueName: \"kubernetes.io/projected/7a810b4d-d7e5-4d1b-8299-71fae1652240-kube-api-access-4cxfn\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:32.996412 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:32.996345 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6f444d8-62e1-4f3f-b7dd-9d31399d7534-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h\" (UID: \"f6f444d8-62e1-4f3f-b7dd-9d31399d7534\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h" Nov 01 06:15:32.996412 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:32.996401 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6mnx\" (UniqueName: \"kubernetes.io/projected/f6f444d8-62e1-4f3f-b7dd-9d31399d7534-kube-api-access-h6mnx\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h\" (UID: \"f6f444d8-62e1-4f3f-b7dd-9d31399d7534\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h" Nov 01 06:15:32.996705 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:32.996449 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f6f444d8-62e1-4f3f-b7dd-9d31399d7534-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h\" (UID: \"f6f444d8-62e1-4f3f-b7dd-9d31399d7534\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h" Nov 01 06:15:32.996705 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:32.996563 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f6f444d8-62e1-4f3f-b7dd-9d31399d7534-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h\" (UID: \"f6f444d8-62e1-4f3f-b7dd-9d31399d7534\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h" Nov 01 06:15:32.997156 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:32.997129 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f6f444d8-62e1-4f3f-b7dd-9d31399d7534-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h\" (UID: \"f6f444d8-62e1-4f3f-b7dd-9d31399d7534\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h" Nov 01 06:15:32.997251 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:32.997227 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6f444d8-62e1-4f3f-b7dd-9d31399d7534-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h\" (UID: \"f6f444d8-62e1-4f3f-b7dd-9d31399d7534\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h" Nov 01 06:15:32.998981 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:32.998963 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f6f444d8-62e1-4f3f-b7dd-9d31399d7534-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h\" (UID: \"f6f444d8-62e1-4f3f-b7dd-9d31399d7534\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h" Nov 01 06:15:33.057992 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:33.057966 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6mnx\" (UniqueName: \"kubernetes.io/projected/f6f444d8-62e1-4f3f-b7dd-9d31399d7534-kube-api-access-h6mnx\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h\" (UID: \"f6f444d8-62e1-4f3f-b7dd-9d31399d7534\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h" Nov 01 06:15:33.059760 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:33.059740 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h" Nov 01 06:15:33.454403 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:33.454367 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h"] Nov 01 06:15:33.455081 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:15:33.455059 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f444d8_62e1_4f3f_b7dd_9d31399d7534.slice/crio-76e272a2afcd3216135aad9767299ef763bd010b9eb297c7095ed6cf36ed10aa WatchSource:0}: Error finding container 76e272a2afcd3216135aad9767299ef763bd010b9eb297c7095ed6cf36ed10aa: Status 404 returned error can't find the container with id 76e272a2afcd3216135aad9767299ef763bd010b9eb297c7095ed6cf36ed10aa Nov 01 06:15:33.487769 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:33.487738 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d25194-klk5n" Nov 01 06:15:33.487938 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:33.487771 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d25194-klk5n" event={"ID":"7a810b4d-d7e5-4d1b-8299-71fae1652240","Type":"ContainerDied","Data":"3c5117af985ce4953ef8d6bc559666afd0a6d05d24b8dce7999c4708e0456714"} Nov 01 06:15:33.487938 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:33.487830 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c5117af985ce4953ef8d6bc559666afd0a6d05d24b8dce7999c4708e0456714" Nov 01 06:15:33.489851 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:33.489825 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h" event={"ID":"f6f444d8-62e1-4f3f-b7dd-9d31399d7534","Type":"ContainerStarted","Data":"76e272a2afcd3216135aad9767299ef763bd010b9eb297c7095ed6cf36ed10aa"} Nov 01 06:15:34.499369 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:34.499335 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h" event={"ID":"f6f444d8-62e1-4f3f-b7dd-9d31399d7534","Type":"ContainerStarted","Data":"73aebf27dcbb750aa51cc323fd99999b834ae423e9746cddcf4c6c5bbfddb513"} Nov 01 06:15:34.499815 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:34.499452 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h" Nov 01 06:15:34.619926 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:34.619872 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h" podStartSLOduration=2.619858673 podStartE2EDuration="2.619858673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:15:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:15:34.617074988 +0000 UTC m=+1813.272192155" watchObservedRunningTime="2026-11-01 06:15:34.619858673 +0000 UTC m=+1813.274975887" Nov 01 06:15:39.974537 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:15:39.974498 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:15:42.480497 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:42.480452 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm" Nov 01 06:15:45.312618 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:45.312571 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33b1bc69-tljnk"] Nov 01 06:15:45.316358 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:45.316337 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b1bc69-tljnk" Nov 01 06:15:45.370409 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:45.370371 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:15:45.370409 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:45.370372 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:15:45.421059 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:45.421020 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:15:45.421059 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:45.421046 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:15:45.424190 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:45.424156 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33b1bc69-tljnk"] Nov 01 06:15:45.432415 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:45.432382 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/709afc4b-6023-48bc-a35b-20f4ba689011-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33b1bc69-tljnk\" (UID: \"709afc4b-6023-48bc-a35b-20f4ba689011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b1bc69-tljnk" Nov 01 06:15:45.432577 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:45.432433 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpgcw\" (UniqueName: \"kubernetes.io/projected/709afc4b-6023-48bc-a35b-20f4ba689011-kube-api-access-zpgcw\") pod \"maas-api-revoke-sub-33b1bc69-tljnk\" (UID: \"709afc4b-6023-48bc-a35b-20f4ba689011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b1bc69-tljnk" Nov 01 06:15:45.432644 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:45.432594 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/709afc4b-6023-48bc-a35b-20f4ba689011-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33b1bc69-tljnk\" (UID: \"709afc4b-6023-48bc-a35b-20f4ba689011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b1bc69-tljnk" Nov 01 06:15:45.515099 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:45.515068 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h" Nov 01 06:15:45.533963 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:45.533934 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/709afc4b-6023-48bc-a35b-20f4ba689011-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33b1bc69-tljnk\" (UID: \"709afc4b-6023-48bc-a35b-20f4ba689011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b1bc69-tljnk" Nov 01 06:15:45.534167 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:45.534017 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/709afc4b-6023-48bc-a35b-20f4ba689011-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33b1bc69-tljnk\" (UID: \"709afc4b-6023-48bc-a35b-20f4ba689011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b1bc69-tljnk" Nov 01 06:15:45.534167 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:45.534044 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpgcw\" (UniqueName: \"kubernetes.io/projected/709afc4b-6023-48bc-a35b-20f4ba689011-kube-api-access-zpgcw\") pod \"maas-api-revoke-sub-33b1bc69-tljnk\" (UID: \"709afc4b-6023-48bc-a35b-20f4ba689011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b1bc69-tljnk" Nov 01 06:15:45.534937 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:45.534910 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/709afc4b-6023-48bc-a35b-20f4ba689011-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33b1bc69-tljnk\" (UID: \"709afc4b-6023-48bc-a35b-20f4ba689011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b1bc69-tljnk" Nov 01 06:15:45.536522 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:45.536471 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/709afc4b-6023-48bc-a35b-20f4ba689011-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33b1bc69-tljnk\" (UID: \"709afc4b-6023-48bc-a35b-20f4ba689011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b1bc69-tljnk" Nov 01 06:15:45.596225 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:45.596149 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpgcw\" (UniqueName: \"kubernetes.io/projected/709afc4b-6023-48bc-a35b-20f4ba689011-kube-api-access-zpgcw\") pod \"maas-api-revoke-sub-33b1bc69-tljnk\" (UID: \"709afc4b-6023-48bc-a35b-20f4ba689011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b1bc69-tljnk" Nov 01 06:15:45.632116 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:45.632085 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b1bc69-tljnk" Nov 01 06:15:46.031568 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:46.031541 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33b1bc69-tljnk"] Nov 01 06:15:46.033275 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:15:46.033239 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod709afc4b_6023_48bc_a35b_20f4ba689011.slice/crio-13e30146dcd920c5c250e4845bc5ec5c296a103fbc711ceaa982b9ef20c1898d WatchSource:0}: Error finding container 13e30146dcd920c5c250e4845bc5ec5c296a103fbc711ceaa982b9ef20c1898d: Status 404 returned error can't find the container with id 13e30146dcd920c5c250e4845bc5ec5c296a103fbc711ceaa982b9ef20c1898d Nov 01 06:15:46.589511 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:46.589386 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b1bc69-tljnk" event={"ID":"709afc4b-6023-48bc-a35b-20f4ba689011","Type":"ContainerStarted","Data":"0c543f791003f05efb5160c4f3d724c3b4b7da2f471195b1e0ad4000afb63bf3"} Nov 01 06:15:46.589511 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:46.589452 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b1bc69-tljnk" event={"ID":"709afc4b-6023-48bc-a35b-20f4ba689011","Type":"ContainerStarted","Data":"13e30146dcd920c5c250e4845bc5ec5c296a103fbc711ceaa982b9ef20c1898d"} Nov 01 06:15:46.708724 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:46.708670 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b1bc69-tljnk" podStartSLOduration=1.708652958 podStartE2EDuration="1.708652958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:15:46.705967024 +0000 UTC m=+1825.361084192" watchObservedRunningTime="2026-11-01 06:15:46.708652958 +0000 UTC m=+1825.363770125" Nov 01 06:15:47.598545 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:47.598507 5040 generic.go:356] "Generic (PLEG): container finished" podID="709afc4b-6023-48bc-a35b-20f4ba689011" containerID="0c543f791003f05efb5160c4f3d724c3b4b7da2f471195b1e0ad4000afb63bf3" exitCode=0 Nov 01 06:15:47.598918 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:47.598587 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b1bc69-tljnk" event={"ID":"709afc4b-6023-48bc-a35b-20f4ba689011","Type":"ContainerDied","Data":"0c543f791003f05efb5160c4f3d724c3b4b7da2f471195b1e0ad4000afb63bf3"} Nov 01 06:15:48.750299 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:48.750276 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b1bc69-tljnk" Nov 01 06:15:48.868910 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:48.868801 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/709afc4b-6023-48bc-a35b-20f4ba689011-kube-api-access-zpgcw\" (UniqueName: \"kubernetes.io/projected/709afc4b-6023-48bc-a35b-20f4ba689011-kube-api-access-zpgcw\") pod \"709afc4b-6023-48bc-a35b-20f4ba689011\" (UID: \"709afc4b-6023-48bc-a35b-20f4ba689011\") " Nov 01 06:15:48.869101 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:48.868932 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/709afc4b-6023-48bc-a35b-20f4ba689011-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/709afc4b-6023-48bc-a35b-20f4ba689011-maas-api-openshift-ca\") pod \"709afc4b-6023-48bc-a35b-20f4ba689011\" (UID: \"709afc4b-6023-48bc-a35b-20f4ba689011\") " Nov 01 06:15:48.869101 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:48.869090 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/709afc4b-6023-48bc-a35b-20f4ba689011-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/709afc4b-6023-48bc-a35b-20f4ba689011-maas-api-xks-ca\") pod \"709afc4b-6023-48bc-a35b-20f4ba689011\" (UID: \"709afc4b-6023-48bc-a35b-20f4ba689011\") " Nov 01 06:15:48.869752 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:48.869717 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/709afc4b-6023-48bc-a35b-20f4ba689011-maas-api-openshift-ca" pod "709afc4b-6023-48bc-a35b-20f4ba689011" (UID: "709afc4b-6023-48bc-a35b-20f4ba689011"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:15:48.874365 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:48.874329 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/709afc4b-6023-48bc-a35b-20f4ba689011-kube-api-access-zpgcw" pod "709afc4b-6023-48bc-a35b-20f4ba689011" (UID: "709afc4b-6023-48bc-a35b-20f4ba689011"). InnerVolumeSpecName "kube-api-access-zpgcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:15:48.874661 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:48.874335 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/709afc4b-6023-48bc-a35b-20f4ba689011-maas-api-xks-ca" pod "709afc4b-6023-48bc-a35b-20f4ba689011" (UID: "709afc4b-6023-48bc-a35b-20f4ba689011"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:15:48.970232 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:48.970194 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/709afc4b-6023-48bc-a35b-20f4ba689011-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:48.970232 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:48.970228 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/709afc4b-6023-48bc-a35b-20f4ba689011-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:48.970232 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:48.970238 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpgcw\" (UniqueName: \"kubernetes.io/projected/709afc4b-6023-48bc-a35b-20f4ba689011-kube-api-access-zpgcw\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:49.614323 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:49.614287 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b1bc69-tljnk" Nov 01 06:15:49.614323 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:49.614292 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b1bc69-tljnk" event={"ID":"709afc4b-6023-48bc-a35b-20f4ba689011","Type":"ContainerDied","Data":"13e30146dcd920c5c250e4845bc5ec5c296a103fbc711ceaa982b9ef20c1898d"} Nov 01 06:15:49.614614 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:49.614344 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13e30146dcd920c5c250e4845bc5ec5c296a103fbc711ceaa982b9ef20c1898d" Nov 01 06:15:54.058388 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.058349 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh"] Nov 01 06:15:54.058865 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.058849 5040 cpu_manager.go:414] "containerMap: removing container" podUID="709afc4b-6023-48bc-a35b-20f4ba689011" containerName="revoke-keys" Nov 01 06:15:54.058916 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.058866 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="709afc4b-6023-48bc-a35b-20f4ba689011" containerName="revoke-keys" Nov 01 06:15:54.058958 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.058947 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="709afc4b-6023-48bc-a35b-20f4ba689011" containerName="revoke-keys" Nov 01 06:15:54.068985 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.068956 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" Nov 01 06:15:54.125295 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.125259 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dab6153a-193e-49d3-b9a4-3aee41f22bf1-workload-certs\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh\" (UID: \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" Nov 01 06:15:54.125295 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.125300 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab6153a-193e-49d3-b9a4-3aee41f22bf1-credential-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh\" (UID: \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" Nov 01 06:15:54.125525 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.125408 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dab6153a-193e-49d3-b9a4-3aee41f22bf1-istio-token\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh\" (UID: \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" Nov 01 06:15:54.125525 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.125442 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab6153a-193e-49d3-b9a4-3aee41f22bf1-workload-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh\" (UID: \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" Nov 01 06:15:54.125525 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.125459 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqcjc\" (UniqueName: \"kubernetes.io/projected/dab6153a-193e-49d3-b9a4-3aee41f22bf1-kube-api-access-mqcjc\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh\" (UID: \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" Nov 01 06:15:54.125647 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.125584 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dab6153a-193e-49d3-b9a4-3aee41f22bf1-istio-podinfo\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh\" (UID: \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" Nov 01 06:15:54.125647 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.125603 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dab6153a-193e-49d3-b9a4-3aee41f22bf1-istiod-ca-cert\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh\" (UID: \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" Nov 01 06:15:54.125721 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.125675 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dab6153a-193e-49d3-b9a4-3aee41f22bf1-istio-envoy\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh\" (UID: \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" Nov 01 06:15:54.125721 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.125711 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dab6153a-193e-49d3-b9a4-3aee41f22bf1-istio-data\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh\" (UID: \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" Nov 01 06:15:54.165599 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.165565 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh"] Nov 01 06:15:54.227241 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.227208 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab6153a-193e-49d3-b9a4-3aee41f22bf1-credential-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh\" (UID: \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" Nov 01 06:15:54.227450 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.227270 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dab6153a-193e-49d3-b9a4-3aee41f22bf1-istio-token\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh\" (UID: \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" Nov 01 06:15:54.227450 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.227291 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab6153a-193e-49d3-b9a4-3aee41f22bf1-workload-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh\" (UID: \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" Nov 01 06:15:54.227450 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.227310 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqcjc\" (UniqueName: \"kubernetes.io/projected/dab6153a-193e-49d3-b9a4-3aee41f22bf1-kube-api-access-mqcjc\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh\" (UID: \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" Nov 01 06:15:54.227450 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.227347 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dab6153a-193e-49d3-b9a4-3aee41f22bf1-istio-podinfo\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh\" (UID: \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" Nov 01 06:15:54.227450 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.227364 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dab6153a-193e-49d3-b9a4-3aee41f22bf1-istiod-ca-cert\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh\" (UID: \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" Nov 01 06:15:54.227766 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.227519 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dab6153a-193e-49d3-b9a4-3aee41f22bf1-istio-envoy\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh\" (UID: \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" Nov 01 06:15:54.227766 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.227586 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dab6153a-193e-49d3-b9a4-3aee41f22bf1-istio-data\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh\" (UID: \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" Nov 01 06:15:54.227766 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.227644 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dab6153a-193e-49d3-b9a4-3aee41f22bf1-workload-certs\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh\" (UID: \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" Nov 01 06:15:54.227766 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.227714 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab6153a-193e-49d3-b9a4-3aee41f22bf1-credential-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh\" (UID: \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" Nov 01 06:15:54.227974 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.227761 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab6153a-193e-49d3-b9a4-3aee41f22bf1-workload-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh\" (UID: \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" Nov 01 06:15:54.228070 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.228047 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dab6153a-193e-49d3-b9a4-3aee41f22bf1-workload-certs\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh\" (UID: \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" Nov 01 06:15:54.228070 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.228061 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dab6153a-193e-49d3-b9a4-3aee41f22bf1-istio-data\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh\" (UID: \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" Nov 01 06:15:54.228154 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.228124 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dab6153a-193e-49d3-b9a4-3aee41f22bf1-istiod-ca-cert\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh\" (UID: \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" Nov 01 06:15:54.229728 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.229711 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dab6153a-193e-49d3-b9a4-3aee41f22bf1-istio-envoy\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh\" (UID: \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" Nov 01 06:15:54.229877 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.229860 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dab6153a-193e-49d3-b9a4-3aee41f22bf1-istio-podinfo\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh\" (UID: \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" Nov 01 06:15:54.289450 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.289424 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dab6153a-193e-49d3-b9a4-3aee41f22bf1-istio-token\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh\" (UID: \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" Nov 01 06:15:54.289687 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.289664 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqcjc\" (UniqueName: \"kubernetes.io/projected/dab6153a-193e-49d3-b9a4-3aee41f22bf1-kube-api-access-mqcjc\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh\" (UID: \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" Nov 01 06:15:54.386178 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.386096 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" Nov 01 06:15:54.820878 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.820851 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh"] Nov 01 06:15:54.822254 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:15:54.822225 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddab6153a_193e_49d3_b9a4_3aee41f22bf1.slice/crio-2473dcb18c9a54f7d0375f33994d7827254e40f623ab55b12a024c904200b817 WatchSource:0}: Error finding container 2473dcb18c9a54f7d0375f33994d7827254e40f623ab55b12a024c904200b817: Status 404 returned error can't find the container with id 2473dcb18c9a54f7d0375f33994d7827254e40f623ab55b12a024c904200b817 Nov 01 06:15:54.824733 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.824698 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:15:54.824842 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.824780 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:15:54.824842 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:54.824824 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:15:55.650575 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:55.650528 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" event={"ID":"dab6153a-193e-49d3-b9a4-3aee41f22bf1","Type":"ContainerStarted","Data":"74badabff435274324d172c70b2728a5fc70138b18215b7af7cc5daf3d11fc12"} Nov 01 06:15:55.650964 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:55.650602 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" event={"ID":"dab6153a-193e-49d3-b9a4-3aee41f22bf1","Type":"ContainerStarted","Data":"2473dcb18c9a54f7d0375f33994d7827254e40f623ab55b12a024c904200b817"} Nov 01 06:15:55.784557 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:55.784502 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" podStartSLOduration=1.784487526 podStartE2EDuration="1.784487526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:15:55.781291912 +0000 UTC m=+1834.436409080" watchObservedRunningTime="2026-11-01 06:15:55.784487526 +0000 UTC m=+1834.439604688" Nov 01 06:15:56.386702 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:56.386665 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" Nov 01 06:15:56.388350 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:56.388312 5040 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 01 06:15:56.388507 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:56.388403 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" podUID="dab6153a-193e-49d3-b9a4-3aee41f22bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 01 06:15:56.912602 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:56.912545 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf"] Nov 01 06:15:56.924776 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:56.923927 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:15:56.956033 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:56.955992 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c87f5c4-df00-4259-823d-b2d51418e42d-workload-socket\") pod \"e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf\" (UID: \"1c87f5c4-df00-4259-823d-b2d51418e42d\") " pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:15:56.956220 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:56.956037 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c87f5c4-df00-4259-823d-b2d51418e42d-istio-data\") pod \"e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf\" (UID: \"1c87f5c4-df00-4259-823d-b2d51418e42d\") " pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:15:56.956220 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:56.956068 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c87f5c4-df00-4259-823d-b2d51418e42d-istiod-ca-cert\") pod \"e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf\" (UID: \"1c87f5c4-df00-4259-823d-b2d51418e42d\") " pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:15:56.956220 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:56.956139 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c87f5c4-df00-4259-823d-b2d51418e42d-credential-socket\") pod \"e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf\" (UID: \"1c87f5c4-df00-4259-823d-b2d51418e42d\") " pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:15:56.956220 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:56.956194 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c87f5c4-df00-4259-823d-b2d51418e42d-istio-envoy\") pod \"e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf\" (UID: \"1c87f5c4-df00-4259-823d-b2d51418e42d\") " pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:15:56.956448 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:56.956285 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c87f5c4-df00-4259-823d-b2d51418e42d-istio-podinfo\") pod \"e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf\" (UID: \"1c87f5c4-df00-4259-823d-b2d51418e42d\") " pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:15:56.956448 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:56.956314 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c87f5c4-df00-4259-823d-b2d51418e42d-workload-certs\") pod \"e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf\" (UID: \"1c87f5c4-df00-4259-823d-b2d51418e42d\") " pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:15:56.956448 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:56.956349 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c87f5c4-df00-4259-823d-b2d51418e42d-istio-token\") pod \"e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf\" (UID: \"1c87f5c4-df00-4259-823d-b2d51418e42d\") " pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:15:56.956448 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:56.956379 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcxzr\" (UniqueName: \"kubernetes.io/projected/1c87f5c4-df00-4259-823d-b2d51418e42d-kube-api-access-fcxzr\") pod \"e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf\" (UID: \"1c87f5c4-df00-4259-823d-b2d51418e42d\") " pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:15:56.981694 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:56.981649 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-71cea4-openshift-default-dockercfg-4sgxs\"" Nov 01 06:15:57.033940 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:57.033902 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf"] Nov 01 06:15:57.057593 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:57.057558 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c87f5c4-df00-4259-823d-b2d51418e42d-workload-socket\") pod \"e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf\" (UID: \"1c87f5c4-df00-4259-823d-b2d51418e42d\") " pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:15:57.057767 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:57.057604 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c87f5c4-df00-4259-823d-b2d51418e42d-istio-data\") pod \"e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf\" (UID: \"1c87f5c4-df00-4259-823d-b2d51418e42d\") " pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:15:57.057767 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:57.057640 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c87f5c4-df00-4259-823d-b2d51418e42d-istiod-ca-cert\") pod \"e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf\" (UID: \"1c87f5c4-df00-4259-823d-b2d51418e42d\") " pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:15:57.057767 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:57.057715 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c87f5c4-df00-4259-823d-b2d51418e42d-credential-socket\") pod \"e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf\" (UID: \"1c87f5c4-df00-4259-823d-b2d51418e42d\") " pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:15:57.057966 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:57.057774 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c87f5c4-df00-4259-823d-b2d51418e42d-istio-envoy\") pod \"e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf\" (UID: \"1c87f5c4-df00-4259-823d-b2d51418e42d\") " pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:15:57.057966 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:57.057857 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c87f5c4-df00-4259-823d-b2d51418e42d-istio-podinfo\") pod \"e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf\" (UID: \"1c87f5c4-df00-4259-823d-b2d51418e42d\") " pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:15:57.057966 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:57.057887 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c87f5c4-df00-4259-823d-b2d51418e42d-workload-certs\") pod \"e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf\" (UID: \"1c87f5c4-df00-4259-823d-b2d51418e42d\") " pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:15:57.057966 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:57.057919 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c87f5c4-df00-4259-823d-b2d51418e42d-istio-token\") pod \"e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf\" (UID: \"1c87f5c4-df00-4259-823d-b2d51418e42d\") " pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:15:57.057966 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:57.057951 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcxzr\" (UniqueName: \"kubernetes.io/projected/1c87f5c4-df00-4259-823d-b2d51418e42d-kube-api-access-fcxzr\") pod \"e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf\" (UID: \"1c87f5c4-df00-4259-823d-b2d51418e42d\") " pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:15:57.058237 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:57.058077 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c87f5c4-df00-4259-823d-b2d51418e42d-istio-data\") pod \"e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf\" (UID: \"1c87f5c4-df00-4259-823d-b2d51418e42d\") " pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:15:57.058237 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:57.058125 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c87f5c4-df00-4259-823d-b2d51418e42d-workload-socket\") pod \"e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf\" (UID: \"1c87f5c4-df00-4259-823d-b2d51418e42d\") " pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:15:57.058237 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:57.058156 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c87f5c4-df00-4259-823d-b2d51418e42d-credential-socket\") pod \"e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf\" (UID: \"1c87f5c4-df00-4259-823d-b2d51418e42d\") " pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:15:57.058554 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:57.058443 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c87f5c4-df00-4259-823d-b2d51418e42d-workload-certs\") pod \"e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf\" (UID: \"1c87f5c4-df00-4259-823d-b2d51418e42d\") " pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:15:57.058554 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:57.058472 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c87f5c4-df00-4259-823d-b2d51418e42d-istiod-ca-cert\") pod \"e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf\" (UID: \"1c87f5c4-df00-4259-823d-b2d51418e42d\") " pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:15:57.060932 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:57.060911 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c87f5c4-df00-4259-823d-b2d51418e42d-istio-envoy\") pod \"e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf\" (UID: \"1c87f5c4-df00-4259-823d-b2d51418e42d\") " pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:15:57.061276 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:57.061260 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c87f5c4-df00-4259-823d-b2d51418e42d-istio-podinfo\") pod \"e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf\" (UID: \"1c87f5c4-df00-4259-823d-b2d51418e42d\") " pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:15:57.123606 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:57.123578 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c87f5c4-df00-4259-823d-b2d51418e42d-istio-token\") pod \"e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf\" (UID: \"1c87f5c4-df00-4259-823d-b2d51418e42d\") " pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:15:57.126382 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:57.126355 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcxzr\" (UniqueName: \"kubernetes.io/projected/1c87f5c4-df00-4259-823d-b2d51418e42d-kube-api-access-fcxzr\") pod \"e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf\" (UID: \"1c87f5c4-df00-4259-823d-b2d51418e42d\") " pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:15:57.241828 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:57.241725 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:15:57.387548 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:57.387348 5040 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 01 06:15:57.387548 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:57.387408 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" podUID="dab6153a-193e-49d3-b9a4-3aee41f22bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 01 06:15:57.724538 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:57.714647 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf"] Nov 01 06:15:57.743406 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:15:57.743375 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c87f5c4_df00_4259_823d_b2d51418e42d.slice/crio-16aad99dcd63246590c1a08248143e13b050390f3fe65cbec42ce1fa4fbcaa16 WatchSource:0}: Error finding container 16aad99dcd63246590c1a08248143e13b050390f3fe65cbec42ce1fa4fbcaa16: Status 404 returned error can't find the container with id 16aad99dcd63246590c1a08248143e13b050390f3fe65cbec42ce1fa4fbcaa16 Nov 01 06:15:57.745809 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:57.745776 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:15:57.745916 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:57.745846 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:15:57.745916 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:57.745875 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:15:58.386895 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:58.386857 5040 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 01 06:15:58.387394 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:58.386922 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" podUID="dab6153a-193e-49d3-b9a4-3aee41f22bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 01 06:15:58.669669 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:58.669575 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" event={"ID":"1c87f5c4-df00-4259-823d-b2d51418e42d","Type":"ContainerStarted","Data":"df427a0991845a5191566cb4608352e8ed40c7af15df647bc01995cf48ba228f"} Nov 01 06:15:58.669669 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:58.669629 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" event={"ID":"1c87f5c4-df00-4259-823d-b2d51418e42d","Type":"ContainerStarted","Data":"16aad99dcd63246590c1a08248143e13b050390f3fe65cbec42ce1fa4fbcaa16"} Nov 01 06:15:58.792765 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:58.792715 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" podStartSLOduration=2.7927019079999997 podStartE2EDuration="2.792701908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:15:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:15:58.789905268 +0000 UTC m=+1837.445022436" watchObservedRunningTime="2026-11-01 06:15:58.792701908 +0000 UTC m=+1837.447819075" Nov 01 06:15:59.242184 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:59.242140 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:15:59.247709 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:59.247680 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:15:59.387004 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:59.386968 5040 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 01 06:15:59.387384 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:59.387020 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" podUID="dab6153a-193e-49d3-b9a4-3aee41f22bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 01 06:15:59.676280 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:59.676250 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:15:59.677195 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:15:59.677172 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:16:00.386938 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:00.386901 5040 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 01 06:16:00.387121 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:00.386957 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" podUID="dab6153a-193e-49d3-b9a4-3aee41f22bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 01 06:16:01.387340 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:01.387299 5040 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 01 06:16:01.387790 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:01.387368 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" podUID="dab6153a-193e-49d3-b9a4-3aee41f22bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 01 06:16:02.390793 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:02.390760 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" Nov 01 06:16:02.393168 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:02.391137 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" Nov 01 06:16:02.393168 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:02.391808 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" Nov 01 06:16:05.269278 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:05.269242 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71cea4-799667f47-zx5bs"] Nov 01 06:16:05.273277 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:05.273259 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71cea4-799667f47-zx5bs" Nov 01 06:16:05.331012 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:05.330980 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71cea4-799667f47-zx5bs"] Nov 01 06:16:05.372226 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:05.372189 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9xrf\" (UniqueName: \"kubernetes.io/projected/232b8e8b-3616-4768-9741-50e93d149e4b-kube-api-access-g9xrf\") pod \"payload-pre-processing-e2e-shared-a-w0-71cea4-799667f47-zx5bs\" (UID: \"232b8e8b-3616-4768-9741-50e93d149e4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71cea4-799667f47-zx5bs" Nov 01 06:16:05.372405 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:05.372262 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/232b8e8b-3616-4768-9741-50e93d149e4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-71cea4-799667f47-zx5bs\" (UID: \"232b8e8b-3616-4768-9741-50e93d149e4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71cea4-799667f47-zx5bs" Nov 01 06:16:05.377465 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:05.377439 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-71cea4\"" Nov 01 06:16:05.377465 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:05.377439 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-71cea4-dockercfg-wrds5\"" Nov 01 06:16:05.437904 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:05.437866 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-71cea4-589654b775-j76mx"] Nov 01 06:16:05.464176 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:05.463829 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71cea4-589654b775-j76mx" Nov 01 06:16:05.473405 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:05.473368 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9xrf\" (UniqueName: \"kubernetes.io/projected/232b8e8b-3616-4768-9741-50e93d149e4b-kube-api-access-g9xrf\") pod \"payload-pre-processing-e2e-shared-a-w0-71cea4-799667f47-zx5bs\" (UID: \"232b8e8b-3616-4768-9741-50e93d149e4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71cea4-799667f47-zx5bs" Nov 01 06:16:05.473711 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:05.473689 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/232b8e8b-3616-4768-9741-50e93d149e4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-71cea4-799667f47-zx5bs\" (UID: \"232b8e8b-3616-4768-9741-50e93d149e4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71cea4-799667f47-zx5bs" Nov 01 06:16:05.475150 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:05.475093 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/232b8e8b-3616-4768-9741-50e93d149e4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-71cea4-799667f47-zx5bs\" (UID: \"232b8e8b-3616-4768-9741-50e93d149e4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71cea4-799667f47-zx5bs" Nov 01 06:16:05.480210 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:05.480180 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-71cea4-589654b775-j76mx"] Nov 01 06:16:05.535866 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:05.535783 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9xrf\" (UniqueName: \"kubernetes.io/projected/232b8e8b-3616-4768-9741-50e93d149e4b-kube-api-access-g9xrf\") pod \"payload-pre-processing-e2e-shared-a-w0-71cea4-799667f47-zx5bs\" (UID: \"232b8e8b-3616-4768-9741-50e93d149e4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71cea4-799667f47-zx5bs" Nov 01 06:16:05.576862 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:05.576817 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttlpt\" (UniqueName: \"kubernetes.io/projected/785f90ff-95cf-4273-9952-6877d4ffb72d-kube-api-access-ttlpt\") pod \"payload-processing-e2e-shared-a-w0-71cea4-589654b775-j76mx\" (UID: \"785f90ff-95cf-4273-9952-6877d4ffb72d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71cea4-589654b775-j76mx" Nov 01 06:16:05.577068 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:05.576992 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/785f90ff-95cf-4273-9952-6877d4ffb72d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-71cea4-589654b775-j76mx\" (UID: \"785f90ff-95cf-4273-9952-6877d4ffb72d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71cea4-589654b775-j76mx" Nov 01 06:16:05.589118 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:05.589084 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71cea4-799667f47-zx5bs" Nov 01 06:16:05.678729 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:05.678168 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttlpt\" (UniqueName: \"kubernetes.io/projected/785f90ff-95cf-4273-9952-6877d4ffb72d-kube-api-access-ttlpt\") pod \"payload-processing-e2e-shared-a-w0-71cea4-589654b775-j76mx\" (UID: \"785f90ff-95cf-4273-9952-6877d4ffb72d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71cea4-589654b775-j76mx" Nov 01 06:16:05.678729 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:05.678240 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/785f90ff-95cf-4273-9952-6877d4ffb72d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-71cea4-589654b775-j76mx\" (UID: \"785f90ff-95cf-4273-9952-6877d4ffb72d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71cea4-589654b775-j76mx" Nov 01 06:16:05.679250 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:05.679210 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/785f90ff-95cf-4273-9952-6877d4ffb72d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-71cea4-589654b775-j76mx\" (UID: \"785f90ff-95cf-4273-9952-6877d4ffb72d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71cea4-589654b775-j76mx" Nov 01 06:16:05.743011 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:05.742973 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttlpt\" (UniqueName: \"kubernetes.io/projected/785f90ff-95cf-4273-9952-6877d4ffb72d-kube-api-access-ttlpt\") pod \"payload-processing-e2e-shared-a-w0-71cea4-589654b775-j76mx\" (UID: \"785f90ff-95cf-4273-9952-6877d4ffb72d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71cea4-589654b775-j76mx" Nov 01 06:16:05.793072 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:05.792986 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71cea4-589654b775-j76mx" Nov 01 06:16:05.968810 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:05.968768 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71cea4-799667f47-zx5bs"] Nov 01 06:16:05.969398 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:16:05.969364 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232b8e8b_3616_4768_9741_50e93d149e4b.slice/crio-a491b05a6c31629566094151595b78bc921f063b4e1a9e36143f000f599f699d WatchSource:0}: Error finding container a491b05a6c31629566094151595b78bc921f063b4e1a9e36143f000f599f699d: Status 404 returned error can't find the container with id a491b05a6c31629566094151595b78bc921f063b4e1a9e36143f000f599f699d Nov 01 06:16:06.180120 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:06.180095 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-71cea4-589654b775-j76mx"] Nov 01 06:16:06.180823 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:16:06.180801 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod785f90ff_95cf_4273_9952_6877d4ffb72d.slice/crio-a602025192242dd1868c7e857e068a65604315020179b541cae912b1bcaec39d WatchSource:0}: Error finding container a602025192242dd1868c7e857e068a65604315020179b541cae912b1bcaec39d: Status 404 returned error can't find the container with id a602025192242dd1868c7e857e068a65604315020179b541cae912b1bcaec39d Nov 01 06:16:06.499981 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:06.499947 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3578ce-7jlxj"] Nov 01 06:16:06.504934 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:06.504906 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3578ce-7jlxj" Nov 01 06:16:06.558053 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:06.558022 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:16:06.558734 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:06.558713 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:16:06.558826 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:06.558804 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:16:06.558881 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:06.558869 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:16:06.559284 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:06.559260 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3578ce-7jlxj"] Nov 01 06:16:06.588113 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:06.588073 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w24g8\" (UniqueName: \"kubernetes.io/projected/6606cd83-2a3b-4e15-97b3-d5f2315079bd-kube-api-access-w24g8\") pod \"maas-api-revoke-keys-e2e-praxis-a-3578ce-7jlxj\" (UID: \"6606cd83-2a3b-4e15-97b3-d5f2315079bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3578ce-7jlxj" Nov 01 06:16:06.588326 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:06.588220 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6606cd83-2a3b-4e15-97b3-d5f2315079bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3578ce-7jlxj\" (UID: \"6606cd83-2a3b-4e15-97b3-d5f2315079bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3578ce-7jlxj" Nov 01 06:16:06.588326 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:06.588311 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6606cd83-2a3b-4e15-97b3-d5f2315079bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3578ce-7jlxj\" (UID: \"6606cd83-2a3b-4e15-97b3-d5f2315079bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3578ce-7jlxj" Nov 01 06:16:06.689293 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:06.689261 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6606cd83-2a3b-4e15-97b3-d5f2315079bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3578ce-7jlxj\" (UID: \"6606cd83-2a3b-4e15-97b3-d5f2315079bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3578ce-7jlxj" Nov 01 06:16:06.689467 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:06.689318 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6606cd83-2a3b-4e15-97b3-d5f2315079bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3578ce-7jlxj\" (UID: \"6606cd83-2a3b-4e15-97b3-d5f2315079bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3578ce-7jlxj" Nov 01 06:16:06.689555 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:06.689523 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w24g8\" (UniqueName: \"kubernetes.io/projected/6606cd83-2a3b-4e15-97b3-d5f2315079bd-kube-api-access-w24g8\") pod \"maas-api-revoke-keys-e2e-praxis-a-3578ce-7jlxj\" (UID: \"6606cd83-2a3b-4e15-97b3-d5f2315079bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3578ce-7jlxj" Nov 01 06:16:06.690098 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:06.690081 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6606cd83-2a3b-4e15-97b3-d5f2315079bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3578ce-7jlxj\" (UID: \"6606cd83-2a3b-4e15-97b3-d5f2315079bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3578ce-7jlxj" Nov 01 06:16:06.691711 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:06.691694 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6606cd83-2a3b-4e15-97b3-d5f2315079bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3578ce-7jlxj\" (UID: \"6606cd83-2a3b-4e15-97b3-d5f2315079bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3578ce-7jlxj" Nov 01 06:16:06.724222 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:06.724181 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71cea4-799667f47-zx5bs" event={"ID":"232b8e8b-3616-4768-9741-50e93d149e4b","Type":"ContainerStarted","Data":"98ca89d7a9974d487c924ddcc9c9299c19d85897a09a2e9a0d55a08a3acd69b2"} Nov 01 06:16:06.724398 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:06.724267 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71cea4-799667f47-zx5bs" Nov 01 06:16:06.724398 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:06.724289 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71cea4-799667f47-zx5bs" event={"ID":"232b8e8b-3616-4768-9741-50e93d149e4b","Type":"ContainerStarted","Data":"a491b05a6c31629566094151595b78bc921f063b4e1a9e36143f000f599f699d"} Nov 01 06:16:06.725935 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:06.725915 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71cea4-589654b775-j76mx" event={"ID":"785f90ff-95cf-4273-9952-6877d4ffb72d","Type":"ContainerStarted","Data":"c35c43d2b8f46576a8609704f55e86afaae337c347947802b1e8b052934badc1"} Nov 01 06:16:06.726011 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:06.725951 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71cea4-589654b775-j76mx" event={"ID":"785f90ff-95cf-4273-9952-6877d4ffb72d","Type":"ContainerStarted","Data":"a602025192242dd1868c7e857e068a65604315020179b541cae912b1bcaec39d"} Nov 01 06:16:06.726011 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:06.725970 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71cea4-589654b775-j76mx" Nov 01 06:16:06.751545 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:06.751474 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w24g8\" (UniqueName: \"kubernetes.io/projected/6606cd83-2a3b-4e15-97b3-d5f2315079bd-kube-api-access-w24g8\") pod \"maas-api-revoke-keys-e2e-praxis-a-3578ce-7jlxj\" (UID: \"6606cd83-2a3b-4e15-97b3-d5f2315079bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3578ce-7jlxj" Nov 01 06:16:06.823901 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:06.823873 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3578ce-7jlxj" Nov 01 06:16:06.846875 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:06.846828 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71cea4-799667f47-zx5bs" podStartSLOduration=1.846813118 podStartE2EDuration="1.846813118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:16:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:16:06.845734895 +0000 UTC m=+1845.500852065" watchObservedRunningTime="2026-11-01 06:16:06.846813118 +0000 UTC m=+1845.501930287" Nov 01 06:16:06.963808 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:06.963749 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71cea4-589654b775-j76mx" podStartSLOduration=1.963734307 podStartE2EDuration="1.963734307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:16:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:16:06.962945132 +0000 UTC m=+1845.618062303" watchObservedRunningTime="2026-11-01 06:16:06.963734307 +0000 UTC m=+1845.618851474" Nov 01 06:16:07.203035 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:07.203010 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3578ce-7jlxj"] Nov 01 06:16:07.203763 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:16:07.203739 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6606cd83_2a3b_4e15_97b3_d5f2315079bd.slice/crio-fdc970830b4788d962316a58e0ede09ddc23649d8448c6d18bd20c8e9f57f4d2 WatchSource:0}: Error finding container fdc970830b4788d962316a58e0ede09ddc23649d8448c6d18bd20c8e9f57f4d2: Status 404 returned error can't find the container with id fdc970830b4788d962316a58e0ede09ddc23649d8448c6d18bd20c8e9f57f4d2 Nov 01 06:16:07.733662 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:07.733623 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3578ce-7jlxj" event={"ID":"6606cd83-2a3b-4e15-97b3-d5f2315079bd","Type":"ContainerStarted","Data":"3524b61296cf9f90bcd66b681339fb4186f974a79909f766bac23349f3a76787"} Nov 01 06:16:07.734119 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:07.733690 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3578ce-7jlxj" event={"ID":"6606cd83-2a3b-4e15-97b3-d5f2315079bd","Type":"ContainerStarted","Data":"fdc970830b4788d962316a58e0ede09ddc23649d8448c6d18bd20c8e9f57f4d2"} Nov 01 06:16:07.851129 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:07.851025 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3578ce-7jlxj" podStartSLOduration=1.851012211 podStartE2EDuration="1.851012211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:16:07.849848429 +0000 UTC m=+1846.504965596" watchObservedRunningTime="2026-11-01 06:16:07.851012211 +0000 UTC m=+1846.506129377" Nov 01 06:16:08.742881 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:08.742842 5040 generic.go:356] "Generic (PLEG): container finished" podID="6606cd83-2a3b-4e15-97b3-d5f2315079bd" containerID="3524b61296cf9f90bcd66b681339fb4186f974a79909f766bac23349f3a76787" exitCode=0 Nov 01 06:16:08.743302 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:08.742916 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3578ce-7jlxj" event={"ID":"6606cd83-2a3b-4e15-97b3-d5f2315079bd","Type":"ContainerDied","Data":"3524b61296cf9f90bcd66b681339fb4186f974a79909f766bac23349f3a76787"} Nov 01 06:16:09.896865 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:09.896836 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3578ce-7jlxj" Nov 01 06:16:09.973738 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:16:09.973702 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:16:10.025179 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:10.025093 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6606cd83-2a3b-4e15-97b3-d5f2315079bd-kube-api-access-w24g8\" (UniqueName: \"kubernetes.io/projected/6606cd83-2a3b-4e15-97b3-d5f2315079bd-kube-api-access-w24g8\") pod \"6606cd83-2a3b-4e15-97b3-d5f2315079bd\" (UID: \"6606cd83-2a3b-4e15-97b3-d5f2315079bd\") " Nov 01 06:16:10.025352 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:10.025208 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6606cd83-2a3b-4e15-97b3-d5f2315079bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6606cd83-2a3b-4e15-97b3-d5f2315079bd-maas-api-xks-ca\") pod \"6606cd83-2a3b-4e15-97b3-d5f2315079bd\" (UID: \"6606cd83-2a3b-4e15-97b3-d5f2315079bd\") " Nov 01 06:16:10.025352 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:10.025316 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6606cd83-2a3b-4e15-97b3-d5f2315079bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6606cd83-2a3b-4e15-97b3-d5f2315079bd-maas-api-openshift-ca\") pod \"6606cd83-2a3b-4e15-97b3-d5f2315079bd\" (UID: \"6606cd83-2a3b-4e15-97b3-d5f2315079bd\") " Nov 01 06:16:10.025731 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:10.025691 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6606cd83-2a3b-4e15-97b3-d5f2315079bd-maas-api-openshift-ca" pod "6606cd83-2a3b-4e15-97b3-d5f2315079bd" (UID: "6606cd83-2a3b-4e15-97b3-d5f2315079bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:16:10.027273 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:10.027245 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6606cd83-2a3b-4e15-97b3-d5f2315079bd-kube-api-access-w24g8" pod "6606cd83-2a3b-4e15-97b3-d5f2315079bd" (UID: "6606cd83-2a3b-4e15-97b3-d5f2315079bd"). InnerVolumeSpecName "kube-api-access-w24g8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:16:10.027727 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:10.027705 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6606cd83-2a3b-4e15-97b3-d5f2315079bd-maas-api-xks-ca" pod "6606cd83-2a3b-4e15-97b3-d5f2315079bd" (UID: "6606cd83-2a3b-4e15-97b3-d5f2315079bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:16:10.126922 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:10.126886 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w24g8\" (UniqueName: \"kubernetes.io/projected/6606cd83-2a3b-4e15-97b3-d5f2315079bd-kube-api-access-w24g8\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:16:10.126922 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:10.126924 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6606cd83-2a3b-4e15-97b3-d5f2315079bd-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:16:10.127126 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:10.126934 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6606cd83-2a3b-4e15-97b3-d5f2315079bd-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:16:10.756252 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:10.756209 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3578ce-7jlxj" event={"ID":"6606cd83-2a3b-4e15-97b3-d5f2315079bd","Type":"ContainerDied","Data":"fdc970830b4788d962316a58e0ede09ddc23649d8448c6d18bd20c8e9f57f4d2"} Nov 01 06:16:10.756252 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:10.756242 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3578ce-7jlxj" Nov 01 06:16:10.756499 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:10.756266 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdc970830b4788d962316a58e0ede09ddc23649d8448c6d18bd20c8e9f57f4d2" Nov 01 06:16:17.736692 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:17.736660 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71cea4-799667f47-zx5bs" Nov 01 06:16:17.737122 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:17.736745 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71cea4-589654b775-j76mx" Nov 01 06:16:22.582261 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:16:22.582226 5040 scope.go:122] "RemoveContainer" containerID="c4e51e91787b5727c54ff2ad477f26a82570c94acddd357612b2d4fa82aa859d" Nov 01 06:16:39.973017 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:16:39.972980 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:17:00.449402 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:17:00.449368 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3578ce-7jlxj"] Nov 01 06:17:00.502001 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:17:00.501959 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3578ce-7jlxj"] Nov 01 06:17:01.514924 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:17:01.514891 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6606cd83-2a3b-4e15-97b3-d5f2315079bd" path="/var/lib/kubelet/pods/6606cd83-2a3b-4e15-97b3-d5f2315079bd/volumes" Nov 01 06:17:09.972135 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:17:09.972092 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:17:22.611083 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:17:22.611048 5040 scope.go:122] "RemoveContainer" containerID="4639b7a60a2bef9706e99765375aaf28c20dec80f94a484151470526c38ccb7b" Nov 01 06:17:39.973702 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:17:39.973617 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:17:45.718674 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:17:45.718627 5040 patch_prober.go:29] interesting pod/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 06:17:45.719174 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:17:45.718701 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" podUID="1c87f5c4-df00-4259-823d-b2d51418e42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 06:17:48.397912 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:17:48.397871 5040 generic.go:356] "Generic (PLEG): container finished" podID="1c87f5c4-df00-4259-823d-b2d51418e42d" containerID="df427a0991845a5191566cb4608352e8ed40c7af15df647bc01995cf48ba228f" exitCode=0 Nov 01 06:17:48.398386 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:17:48.397947 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" event={"ID":"1c87f5c4-df00-4259-823d-b2d51418e42d","Type":"ContainerDied","Data":"df427a0991845a5191566cb4608352e8ed40c7af15df647bc01995cf48ba228f"} Nov 01 06:17:48.398386 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:17:48.398377 5040 scope.go:122] "RemoveContainer" containerID="df427a0991845a5191566cb4608352e8ed40c7af15df647bc01995cf48ba228f" Nov 01 06:17:48.399886 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:17:48.399855 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:17:48.399987 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:17:48.399923 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:17:48.399987 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:17:48.399960 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:17:49.425061 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:17:49.425023 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" event={"ID":"1c87f5c4-df00-4259-823d-b2d51418e42d","Type":"ContainerStarted","Data":"3eef6d7c23966931820cdbdef29c4123e74b60848d7585a09f8333b0e3f8724d"} Nov 01 06:17:50.242710 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:17:50.242666 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:17:50.242916 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:17:50.242855 5040 patch_prober.go:29] interesting pod/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 01 06:17:50.242916 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:17:50.242901 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" podUID="1c87f5c4-df00-4259-823d-b2d51418e42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 01 06:17:51.242411 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:17:51.242375 5040 patch_prober.go:29] interesting pod/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 01 06:17:51.242922 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:17:51.242438 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" podUID="1c87f5c4-df00-4259-823d-b2d51418e42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 01 06:17:52.242285 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:17:52.242249 5040 patch_prober.go:29] interesting pod/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 01 06:17:52.242538 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:17:52.242314 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" podUID="1c87f5c4-df00-4259-823d-b2d51418e42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 01 06:17:53.242834 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:17:53.242804 5040 patch_prober.go:29] interesting pod/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 01 06:17:53.243217 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:17:53.242854 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" podUID="1c87f5c4-df00-4259-823d-b2d51418e42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 01 06:17:54.243025 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:17:54.242986 5040 patch_prober.go:29] interesting pod/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 01 06:17:54.243451 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:17:54.243042 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" podUID="1c87f5c4-df00-4259-823d-b2d51418e42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 01 06:17:55.246111 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:17:55.246079 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:17:55.246625 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:17:55.246456 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:17:55.247249 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:17:55.247230 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:18:09.972545 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:18:09.972475 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:18:25.778370 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:25.778333 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44"] Nov 01 06:18:25.779126 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:25.778648 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" podUID="6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb" containerName="istio-proxy" containerID="cri-o://b9d9a0e02f39fa4413b97efdb62c664305162a959ce91cce1390f9b0adcf1108" gracePeriod=30 Nov 01 06:18:31.042308 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:31.042245 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" Nov 01 06:18:31.180743 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:31.180707 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-istio-envoy\") pod \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\" (UID: \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\") " Nov 01 06:18:31.180743 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:31.180746 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-istio-data\") pod \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\" (UID: \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\") " Nov 01 06:18:31.181008 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:31.180827 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-credential-socket\") pod \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\" (UID: \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\") " Nov 01 06:18:31.181008 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:31.180846 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-istio-token\" (UniqueName: \"kubernetes.io/projected/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-istio-token\") pod \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\" (UID: \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\") " Nov 01 06:18:31.181008 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:31.180865 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-istio-podinfo\") pod \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\" (UID: \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\") " Nov 01 06:18:31.181008 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:31.180899 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-istiod-ca-cert\") pod \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\" (UID: \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\") " Nov 01 06:18:31.181008 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:31.180976 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-workload-socket\") pod \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\" (UID: \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\") " Nov 01 06:18:31.181269 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:31.181041 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-kube-api-access-6zznt\" (UniqueName: \"kubernetes.io/projected/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-kube-api-access-6zznt\") pod \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\" (UID: \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\") " Nov 01 06:18:31.181269 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:31.181072 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-workload-certs\") pod \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\" (UID: \"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb\") " Nov 01 06:18:31.181269 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:31.181207 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-credential-socket" pod "6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb" (UID: "6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:18:31.181549 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:31.181397 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-workload-socket" pod "6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb" (UID: "6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:18:31.181698 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:31.181460 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-workload-certs" pod "6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb" (UID: "6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:18:31.181698 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:31.181222 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-istio-data" pod "6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb" (UID: "6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:18:31.181698 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:31.181586 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-istiod-ca-cert" pod "6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb" (UID: "6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:18:31.181698 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:31.181658 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:18:31.181698 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:31.181678 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:18:31.181698 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:31.181694 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:18:31.182057 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:31.181708 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:18:31.183144 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:31.183118 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-istio-envoy" pod "6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb" (UID: "6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:18:31.183282 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:31.183254 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-istio-podinfo" pod "6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb" (UID: "6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:18:31.183374 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:31.183280 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-istio-token" pod "6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb" (UID: "6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:18:31.183374 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:31.183320 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-kube-api-access-6zznt" pod "6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb" (UID: "6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb"). InnerVolumeSpecName "kube-api-access-6zznt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:18:31.282864 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:31.282832 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:18:31.282864 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:31.282861 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:18:31.282864 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:31.282872 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:18:31.283172 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:31.282881 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zznt\" (UniqueName: \"kubernetes.io/projected/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-kube-api-access-6zznt\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:18:31.283172 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:31.282891 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:18:31.688441 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:31.688404 5040 generic.go:356] "Generic (PLEG): container finished" podID="6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb" containerID="b9d9a0e02f39fa4413b97efdb62c664305162a959ce91cce1390f9b0adcf1108" exitCode=0 Nov 01 06:18:31.688667 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:31.688447 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" Nov 01 06:18:31.688667 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:31.688507 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" event={"ID":"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb","Type":"ContainerDied","Data":"b9d9a0e02f39fa4413b97efdb62c664305162a959ce91cce1390f9b0adcf1108"} Nov 01 06:18:31.688667 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:31.688568 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44" event={"ID":"6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb","Type":"ContainerDied","Data":"0d51ad36cec04a8369ed22d7243670c07e7522ac3f421cabe53ad17e4795c5dc"} Nov 01 06:18:31.688667 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:31.688600 5040 scope.go:122] "RemoveContainer" containerID="b9d9a0e02f39fa4413b97efdb62c664305162a959ce91cce1390f9b0adcf1108" Nov 01 06:18:31.698939 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:31.698916 5040 scope.go:122] "RemoveContainer" containerID="b9d9a0e02f39fa4413b97efdb62c664305162a959ce91cce1390f9b0adcf1108" Nov 01 06:18:31.699250 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:18:31.699229 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9d9a0e02f39fa4413b97efdb62c664305162a959ce91cce1390f9b0adcf1108\": container with ID starting with b9d9a0e02f39fa4413b97efdb62c664305162a959ce91cce1390f9b0adcf1108 not found: ID does not exist" containerID="b9d9a0e02f39fa4413b97efdb62c664305162a959ce91cce1390f9b0adcf1108" Nov 01 06:18:31.699295 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:31.699261 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9d9a0e02f39fa4413b97efdb62c664305162a959ce91cce1390f9b0adcf1108"} err="failed to get container status \"b9d9a0e02f39fa4413b97efdb62c664305162a959ce91cce1390f9b0adcf1108\": rpc error: code = NotFound desc = could not find container \"b9d9a0e02f39fa4413b97efdb62c664305162a959ce91cce1390f9b0adcf1108\": container with ID starting with b9d9a0e02f39fa4413b97efdb62c664305162a959ce91cce1390f9b0adcf1108 not found: ID does not exist" Nov 01 06:18:31.906445 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:31.906363 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44"] Nov 01 06:18:31.956580 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:31.956542 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tmf44"] Nov 01 06:18:33.513798 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:18:33.513765 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb" path="/var/lib/kubelet/pods/6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb/volumes" Nov 01 06:18:39.972815 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:18:39.972773 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:19:09.971892 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:19:09.971852 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:19:27.047665 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:27.047602 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1ac44e5-xghg9"] Nov 01 06:19:27.050623 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:27.050604 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1ac44e5-xghg9"] Nov 01 06:19:27.515589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:27.515554 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff" path="/var/lib/kubelet/pods/6bbd5773-34f4-4fb2-b2be-ced0b62ca0ff/volumes" Nov 01 06:19:39.340767 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:39.340728 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8315d578-v4s9v"] Nov 01 06:19:39.341381 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:39.341359 5040 cpu_manager.go:414] "containerMap: removing container" podUID="6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb" containerName="istio-proxy" Nov 01 06:19:39.341509 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:39.341384 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb" containerName="istio-proxy" Nov 01 06:19:39.341509 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:39.341460 5040 cpu_manager.go:414] "containerMap: removing container" podUID="6606cd83-2a3b-4e15-97b3-d5f2315079bd" containerName="revoke-keys" Nov 01 06:19:39.341509 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:39.341470 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6606cd83-2a3b-4e15-97b3-d5f2315079bd" containerName="revoke-keys" Nov 01 06:19:39.341673 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:39.341636 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ea5cbab-4aa1-431f-a4a4-64a2f324e3cb" containerName="istio-proxy" Nov 01 06:19:39.341673 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:39.341661 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="6606cd83-2a3b-4e15-97b3-d5f2315079bd" containerName="revoke-keys" Nov 01 06:19:39.346231 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:39.346210 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8315d578-v4s9v" Nov 01 06:19:39.402140 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:39.402105 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8315d578-v4s9v"] Nov 01 06:19:39.448551 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:39.448514 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:19:39.448551 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:39.448537 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:19:39.448551 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:39.448512 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:19:39.449373 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:39.449357 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:19:39.467902 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:39.467877 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c42be41c-0e3d-41a1-be66-b2a8cf6f8940-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8315d578-v4s9v\" (UID: \"c42be41c-0e3d-41a1-be66-b2a8cf6f8940\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8315d578-v4s9v" Nov 01 06:19:39.468010 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:39.467918 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c42be41c-0e3d-41a1-be66-b2a8cf6f8940-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8315d578-v4s9v\" (UID: \"c42be41c-0e3d-41a1-be66-b2a8cf6f8940\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8315d578-v4s9v" Nov 01 06:19:39.468056 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:39.468038 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h78z2\" (UniqueName: \"kubernetes.io/projected/c42be41c-0e3d-41a1-be66-b2a8cf6f8940-kube-api-access-h78z2\") pod \"maas-api-revoke-sub-8315d578-v4s9v\" (UID: \"c42be41c-0e3d-41a1-be66-b2a8cf6f8940\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8315d578-v4s9v" Nov 01 06:19:39.569598 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:39.569549 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h78z2\" (UniqueName: \"kubernetes.io/projected/c42be41c-0e3d-41a1-be66-b2a8cf6f8940-kube-api-access-h78z2\") pod \"maas-api-revoke-sub-8315d578-v4s9v\" (UID: \"c42be41c-0e3d-41a1-be66-b2a8cf6f8940\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8315d578-v4s9v" Nov 01 06:19:39.569936 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:39.569906 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c42be41c-0e3d-41a1-be66-b2a8cf6f8940-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8315d578-v4s9v\" (UID: \"c42be41c-0e3d-41a1-be66-b2a8cf6f8940\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8315d578-v4s9v" Nov 01 06:19:39.570221 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:39.570195 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c42be41c-0e3d-41a1-be66-b2a8cf6f8940-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8315d578-v4s9v\" (UID: \"c42be41c-0e3d-41a1-be66-b2a8cf6f8940\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8315d578-v4s9v" Nov 01 06:19:39.570842 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:39.570820 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c42be41c-0e3d-41a1-be66-b2a8cf6f8940-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8315d578-v4s9v\" (UID: \"c42be41c-0e3d-41a1-be66-b2a8cf6f8940\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8315d578-v4s9v" Nov 01 06:19:39.572435 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:39.572408 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c42be41c-0e3d-41a1-be66-b2a8cf6f8940-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8315d578-v4s9v\" (UID: \"c42be41c-0e3d-41a1-be66-b2a8cf6f8940\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8315d578-v4s9v" Nov 01 06:19:39.631297 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:39.631225 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h78z2\" (UniqueName: \"kubernetes.io/projected/c42be41c-0e3d-41a1-be66-b2a8cf6f8940-kube-api-access-h78z2\") pod \"maas-api-revoke-sub-8315d578-v4s9v\" (UID: \"c42be41c-0e3d-41a1-be66-b2a8cf6f8940\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8315d578-v4s9v" Nov 01 06:19:39.659246 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:39.659217 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8315d578-v4s9v" Nov 01 06:19:39.972941 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:19:39.972853 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:19:40.039010 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:40.038977 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8315d578-v4s9v"] Nov 01 06:19:40.039818 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:19:40.039791 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc42be41c_0e3d_41a1_be66_b2a8cf6f8940.slice/crio-5dc939ba4afab9a4ef32cda23f2be033bd5112efa53709ea07819e7b2d45bfc8 WatchSource:0}: Error finding container 5dc939ba4afab9a4ef32cda23f2be033bd5112efa53709ea07819e7b2d45bfc8: Status 404 returned error can't find the container with id 5dc939ba4afab9a4ef32cda23f2be033bd5112efa53709ea07819e7b2d45bfc8 Nov 01 06:19:40.138634 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:40.138594 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8315d578-v4s9v" event={"ID":"c42be41c-0e3d-41a1-be66-b2a8cf6f8940","Type":"ContainerStarted","Data":"5dc939ba4afab9a4ef32cda23f2be033bd5112efa53709ea07819e7b2d45bfc8"} Nov 01 06:19:41.147277 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:41.147183 5040 generic.go:356] "Generic (PLEG): container finished" podID="c42be41c-0e3d-41a1-be66-b2a8cf6f8940" containerID="50c0f92716c413352634aa4d7792601c030f4e7aa05e3fe94f95c328efd0d7af" exitCode=0 Nov 01 06:19:41.147972 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:41.147275 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8315d578-v4s9v" event={"ID":"c42be41c-0e3d-41a1-be66-b2a8cf6f8940","Type":"ContainerDied","Data":"50c0f92716c413352634aa4d7792601c030f4e7aa05e3fe94f95c328efd0d7af"} Nov 01 06:19:42.308550 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:42.308517 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8315d578-v4s9v" Nov 01 06:19:42.399565 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:42.399533 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c42be41c-0e3d-41a1-be66-b2a8cf6f8940-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c42be41c-0e3d-41a1-be66-b2a8cf6f8940-maas-api-xks-ca\") pod \"c42be41c-0e3d-41a1-be66-b2a8cf6f8940\" (UID: \"c42be41c-0e3d-41a1-be66-b2a8cf6f8940\") " Nov 01 06:19:42.399785 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:42.399661 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c42be41c-0e3d-41a1-be66-b2a8cf6f8940-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c42be41c-0e3d-41a1-be66-b2a8cf6f8940-maas-api-openshift-ca\") pod \"c42be41c-0e3d-41a1-be66-b2a8cf6f8940\" (UID: \"c42be41c-0e3d-41a1-be66-b2a8cf6f8940\") " Nov 01 06:19:42.399785 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:42.399708 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c42be41c-0e3d-41a1-be66-b2a8cf6f8940-kube-api-access-h78z2\" (UniqueName: \"kubernetes.io/projected/c42be41c-0e3d-41a1-be66-b2a8cf6f8940-kube-api-access-h78z2\") pod \"c42be41c-0e3d-41a1-be66-b2a8cf6f8940\" (UID: \"c42be41c-0e3d-41a1-be66-b2a8cf6f8940\") " Nov 01 06:19:42.400067 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:42.400039 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c42be41c-0e3d-41a1-be66-b2a8cf6f8940-maas-api-openshift-ca" pod "c42be41c-0e3d-41a1-be66-b2a8cf6f8940" (UID: "c42be41c-0e3d-41a1-be66-b2a8cf6f8940"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:19:42.401770 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:42.401744 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c42be41c-0e3d-41a1-be66-b2a8cf6f8940-maas-api-xks-ca" pod "c42be41c-0e3d-41a1-be66-b2a8cf6f8940" (UID: "c42be41c-0e3d-41a1-be66-b2a8cf6f8940"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:19:42.401859 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:42.401771 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c42be41c-0e3d-41a1-be66-b2a8cf6f8940-kube-api-access-h78z2" pod "c42be41c-0e3d-41a1-be66-b2a8cf6f8940" (UID: "c42be41c-0e3d-41a1-be66-b2a8cf6f8940"). InnerVolumeSpecName "kube-api-access-h78z2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:19:42.501056 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:42.501022 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c42be41c-0e3d-41a1-be66-b2a8cf6f8940-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:19:42.501056 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:42.501053 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c42be41c-0e3d-41a1-be66-b2a8cf6f8940-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:19:42.501272 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:42.501067 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h78z2\" (UniqueName: \"kubernetes.io/projected/c42be41c-0e3d-41a1-be66-b2a8cf6f8940-kube-api-access-h78z2\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:19:43.159978 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:43.159949 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8315d578-v4s9v" Nov 01 06:19:43.160160 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:43.159955 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8315d578-v4s9v" event={"ID":"c42be41c-0e3d-41a1-be66-b2a8cf6f8940","Type":"ContainerDied","Data":"5dc939ba4afab9a4ef32cda23f2be033bd5112efa53709ea07819e7b2d45bfc8"} Nov 01 06:19:43.160160 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:43.160062 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5dc939ba4afab9a4ef32cda23f2be033bd5112efa53709ea07819e7b2d45bfc8" Nov 01 06:19:49.705863 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:49.705828 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52"] Nov 01 06:19:49.706311 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:49.706298 5040 cpu_manager.go:414] "containerMap: removing container" podUID="c42be41c-0e3d-41a1-be66-b2a8cf6f8940" containerName="revoke-keys" Nov 01 06:19:49.706355 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:49.706312 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c42be41c-0e3d-41a1-be66-b2a8cf6f8940" containerName="revoke-keys" Nov 01 06:19:49.706406 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:49.706397 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="c42be41c-0e3d-41a1-be66-b2a8cf6f8940" containerName="revoke-keys" Nov 01 06:19:49.709642 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:49.709622 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52" Nov 01 06:19:49.767253 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:49.767217 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52"] Nov 01 06:19:49.779233 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:49.779201 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4wlk\" (UniqueName: \"kubernetes.io/projected/534945c2-d939-41cb-8125-dc2b4b788e40-kube-api-access-f4wlk\") pod \"auth-test-model-71cea4-kserve-5ccbfd498f-b4t52\" (UID: \"534945c2-d939-41cb-8125-dc2b4b788e40\") " pod="ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52" Nov 01 06:19:49.779436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:49.779264 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/534945c2-d939-41cb-8125-dc2b4b788e40-home\") pod \"auth-test-model-71cea4-kserve-5ccbfd498f-b4t52\" (UID: \"534945c2-d939-41cb-8125-dc2b4b788e40\") " pod="ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52" Nov 01 06:19:49.779436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:49.779291 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/534945c2-d939-41cb-8125-dc2b4b788e40-dshm\") pod \"auth-test-model-71cea4-kserve-5ccbfd498f-b4t52\" (UID: \"534945c2-d939-41cb-8125-dc2b4b788e40\") " pod="ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52" Nov 01 06:19:49.779436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:49.779413 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/534945c2-d939-41cb-8125-dc2b4b788e40-tmp-dir\") pod \"auth-test-model-71cea4-kserve-5ccbfd498f-b4t52\" (UID: \"534945c2-d939-41cb-8125-dc2b4b788e40\") " pod="ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52" Nov 01 06:19:49.779669 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:49.779527 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/534945c2-d939-41cb-8125-dc2b4b788e40-tls-certs\") pod \"auth-test-model-71cea4-kserve-5ccbfd498f-b4t52\" (UID: \"534945c2-d939-41cb-8125-dc2b4b788e40\") " pod="ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52" Nov 01 06:19:49.779669 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:49.779560 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/534945c2-d939-41cb-8125-dc2b4b788e40-model-cache\") pod \"auth-test-model-71cea4-kserve-5ccbfd498f-b4t52\" (UID: \"534945c2-d939-41cb-8125-dc2b4b788e40\") " pod="ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52" Nov 01 06:19:49.800658 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:49.800627 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-71cea4\"/\"kube-root-ca.crt\"" Nov 01 06:19:49.800658 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:49.800654 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-71cea4\"/\"openshift-service-ca.crt\"" Nov 01 06:19:49.814131 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:49.814110 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-71cea4\"/\"auth-test-model-71cea4-kserve-self-signed-certs\"" Nov 01 06:19:49.814131 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:49.814113 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-71cea4\"/\"default-dockercfg-sx7gr\"" Nov 01 06:19:49.880709 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:49.880673 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/534945c2-d939-41cb-8125-dc2b4b788e40-model-cache\") pod \"auth-test-model-71cea4-kserve-5ccbfd498f-b4t52\" (UID: \"534945c2-d939-41cb-8125-dc2b4b788e40\") " pod="ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52" Nov 01 06:19:49.880914 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:49.880732 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4wlk\" (UniqueName: \"kubernetes.io/projected/534945c2-d939-41cb-8125-dc2b4b788e40-kube-api-access-f4wlk\") pod \"auth-test-model-71cea4-kserve-5ccbfd498f-b4t52\" (UID: \"534945c2-d939-41cb-8125-dc2b4b788e40\") " pod="ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52" Nov 01 06:19:49.880914 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:49.880778 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/534945c2-d939-41cb-8125-dc2b4b788e40-home\") pod \"auth-test-model-71cea4-kserve-5ccbfd498f-b4t52\" (UID: \"534945c2-d939-41cb-8125-dc2b4b788e40\") " pod="ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52" Nov 01 06:19:49.880914 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:49.880800 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/534945c2-d939-41cb-8125-dc2b4b788e40-dshm\") pod \"auth-test-model-71cea4-kserve-5ccbfd498f-b4t52\" (UID: \"534945c2-d939-41cb-8125-dc2b4b788e40\") " pod="ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52" Nov 01 06:19:49.880914 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:49.880825 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/534945c2-d939-41cb-8125-dc2b4b788e40-tmp-dir\") pod \"auth-test-model-71cea4-kserve-5ccbfd498f-b4t52\" (UID: \"534945c2-d939-41cb-8125-dc2b4b788e40\") " pod="ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52" Nov 01 06:19:49.880914 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:49.880885 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/534945c2-d939-41cb-8125-dc2b4b788e40-tls-certs\") pod \"auth-test-model-71cea4-kserve-5ccbfd498f-b4t52\" (UID: \"534945c2-d939-41cb-8125-dc2b4b788e40\") " pod="ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52" Nov 01 06:19:49.881205 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:49.881101 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/534945c2-d939-41cb-8125-dc2b4b788e40-model-cache\") pod \"auth-test-model-71cea4-kserve-5ccbfd498f-b4t52\" (UID: \"534945c2-d939-41cb-8125-dc2b4b788e40\") " pod="ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52" Nov 01 06:19:49.881205 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:49.881183 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/534945c2-d939-41cb-8125-dc2b4b788e40-home\") pod \"auth-test-model-71cea4-kserve-5ccbfd498f-b4t52\" (UID: \"534945c2-d939-41cb-8125-dc2b4b788e40\") " pod="ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52" Nov 01 06:19:49.881304 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:49.881280 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/534945c2-d939-41cb-8125-dc2b4b788e40-tmp-dir\") pod \"auth-test-model-71cea4-kserve-5ccbfd498f-b4t52\" (UID: \"534945c2-d939-41cb-8125-dc2b4b788e40\") " pod="ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52" Nov 01 06:19:49.883077 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:49.883056 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/534945c2-d939-41cb-8125-dc2b4b788e40-dshm\") pod \"auth-test-model-71cea4-kserve-5ccbfd498f-b4t52\" (UID: \"534945c2-d939-41cb-8125-dc2b4b788e40\") " pod="ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52" Nov 01 06:19:49.883493 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:49.883457 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/534945c2-d939-41cb-8125-dc2b4b788e40-tls-certs\") pod \"auth-test-model-71cea4-kserve-5ccbfd498f-b4t52\" (UID: \"534945c2-d939-41cb-8125-dc2b4b788e40\") " pod="ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52" Nov 01 06:19:49.942700 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:49.942664 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4wlk\" (UniqueName: \"kubernetes.io/projected/534945c2-d939-41cb-8125-dc2b4b788e40-kube-api-access-f4wlk\") pod \"auth-test-model-71cea4-kserve-5ccbfd498f-b4t52\" (UID: \"534945c2-d939-41cb-8125-dc2b4b788e40\") " pod="ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52" Nov 01 06:19:50.022411 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:50.022367 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52" Nov 01 06:19:50.416989 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:50.416949 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52"] Nov 01 06:19:51.048000 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:51.047962 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4dad42b-kfmjj"] Nov 01 06:19:51.050776 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:51.050751 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4dad42b-kfmjj"] Nov 01 06:19:51.214969 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:51.214929 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52" event={"ID":"534945c2-d939-41cb-8125-dc2b4b788e40","Type":"ContainerStarted","Data":"aedc9e828b59fd1df73dff29e8a217a438cc0d03ca2d5a579e97d3f58a7b215d"} Nov 01 06:19:51.215139 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:51.214986 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52" event={"ID":"534945c2-d939-41cb-8125-dc2b4b788e40","Type":"ContainerStarted","Data":"b8f28e78d3f440cdcdc05a516c1d6e07a219603a2485701c1edc1f72f5cd3a15"} Nov 01 06:19:51.335539 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:51.335432 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52" podStartSLOduration=2.335417516 podStartE2EDuration="2.335417516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:19:51.333098335 +0000 UTC m=+2069.988215502" watchObservedRunningTime="2026-11-01 06:19:51.335417516 +0000 UTC m=+2069.990534684" Nov 01 06:19:51.516307 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:51.516274 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1ed0d77-6c9a-492c-9c07-20019eb15284" path="/var/lib/kubelet/pods/a1ed0d77-6c9a-492c-9c07-20019eb15284/volumes" Nov 01 06:19:52.023427 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:52.023390 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52" Nov 01 06:19:54.647559 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:54.647517 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm"] Nov 01 06:19:55.596563 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:55.596526 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-cd78f1/e2e-usage-sim-54291c-7c768bf864-pjszd"] Nov 01 06:19:55.600420 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:55.600400 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-cd78f1/e2e-usage-sim-54291c-7c768bf864-pjszd" Nov 01 06:19:55.654080 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:55.654039 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-cd78f1/e2e-usage-sim-54291c-7c768bf864-pjszd"] Nov 01 06:19:55.655291 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:55.655253 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-cd78f1\"/\"kube-root-ca.crt\"" Nov 01 06:19:55.701772 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:55.701728 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-cd78f1\"/\"default-dockercfg-w5nm6\"" Nov 01 06:19:55.701772 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:55.701728 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-cd78f1\"/\"e2e-usage-sim-54291c-cert\"" Nov 01 06:19:55.701994 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:55.701738 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-cd78f1\"/\"openshift-service-ca.crt\"" Nov 01 06:19:55.743560 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:55.743521 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6e0b1e48-0511-4ef7-8f73-1106983b5b33-tls\") pod \"e2e-usage-sim-54291c-7c768bf864-pjszd\" (UID: \"6e0b1e48-0511-4ef7-8f73-1106983b5b33\") " pod="e2e-extmod-shr-user-cd78f1/e2e-usage-sim-54291c-7c768bf864-pjszd" Nov 01 06:19:55.743719 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:55.743665 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzkd7\" (UniqueName: \"kubernetes.io/projected/6e0b1e48-0511-4ef7-8f73-1106983b5b33-kube-api-access-lzkd7\") pod \"e2e-usage-sim-54291c-7c768bf864-pjszd\" (UID: \"6e0b1e48-0511-4ef7-8f73-1106983b5b33\") " pod="e2e-extmod-shr-user-cd78f1/e2e-usage-sim-54291c-7c768bf864-pjszd" Nov 01 06:19:55.844882 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:55.844848 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6e0b1e48-0511-4ef7-8f73-1106983b5b33-tls\") pod \"e2e-usage-sim-54291c-7c768bf864-pjszd\" (UID: \"6e0b1e48-0511-4ef7-8f73-1106983b5b33\") " pod="e2e-extmod-shr-user-cd78f1/e2e-usage-sim-54291c-7c768bf864-pjszd" Nov 01 06:19:55.845072 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:55.844918 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzkd7\" (UniqueName: \"kubernetes.io/projected/6e0b1e48-0511-4ef7-8f73-1106983b5b33-kube-api-access-lzkd7\") pod \"e2e-usage-sim-54291c-7c768bf864-pjszd\" (UID: \"6e0b1e48-0511-4ef7-8f73-1106983b5b33\") " pod="e2e-extmod-shr-user-cd78f1/e2e-usage-sim-54291c-7c768bf864-pjszd" Nov 01 06:19:55.847378 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:55.847317 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6e0b1e48-0511-4ef7-8f73-1106983b5b33-tls\") pod \"e2e-usage-sim-54291c-7c768bf864-pjszd\" (UID: \"6e0b1e48-0511-4ef7-8f73-1106983b5b33\") " pod="e2e-extmod-shr-user-cd78f1/e2e-usage-sim-54291c-7c768bf864-pjszd" Nov 01 06:19:55.907082 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:55.907047 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzkd7\" (UniqueName: \"kubernetes.io/projected/6e0b1e48-0511-4ef7-8f73-1106983b5b33-kube-api-access-lzkd7\") pod \"e2e-usage-sim-54291c-7c768bf864-pjszd\" (UID: \"6e0b1e48-0511-4ef7-8f73-1106983b5b33\") " pod="e2e-extmod-shr-user-cd78f1/e2e-usage-sim-54291c-7c768bf864-pjszd" Nov 01 06:19:55.912704 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:55.912679 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-cd78f1/e2e-usage-sim-54291c-7c768bf864-pjszd" Nov 01 06:19:56.285620 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:56.285588 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-cd78f1/e2e-usage-sim-54291c-7c768bf864-pjszd"] Nov 01 06:19:56.287208 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:19:56.287179 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e0b1e48_0511_4ef7_8f73_1106983b5b33.slice/crio-d5c0b5566c523f1455424b834800a788c0790b9732ea7e694e2a548e33a8daf1 WatchSource:0}: Error finding container d5c0b5566c523f1455424b834800a788c0790b9732ea7e694e2a548e33a8daf1: Status 404 returned error can't find the container with id d5c0b5566c523f1455424b834800a788c0790b9732ea7e694e2a548e33a8daf1 Nov 01 06:19:56.467946 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:56.467904 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj"] Nov 01 06:19:56.471805 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:56.471784 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" Nov 01 06:19:56.553129 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:56.553048 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e6f829a-4773-4740-a9e4-328091ee920f-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-pnbfj\" (UID: \"9e6f829a-4773-4740-a9e4-328091ee920f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" Nov 01 06:19:56.553129 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:56.553093 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e6f829a-4773-4740-a9e4-328091ee920f-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-pnbfj\" (UID: \"9e6f829a-4773-4740-a9e4-328091ee920f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" Nov 01 06:19:56.553129 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:56.553117 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e6f829a-4773-4740-a9e4-328091ee920f-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-pnbfj\" (UID: \"9e6f829a-4773-4740-a9e4-328091ee920f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" Nov 01 06:19:56.553404 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:56.553248 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e6f829a-4773-4740-a9e4-328091ee920f-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-pnbfj\" (UID: \"9e6f829a-4773-4740-a9e4-328091ee920f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" Nov 01 06:19:56.553404 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:56.553314 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e6f829a-4773-4740-a9e4-328091ee920f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-pnbfj\" (UID: \"9e6f829a-4773-4740-a9e4-328091ee920f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" Nov 01 06:19:56.553404 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:56.553369 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e6f829a-4773-4740-a9e4-328091ee920f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-pnbfj\" (UID: \"9e6f829a-4773-4740-a9e4-328091ee920f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" Nov 01 06:19:56.553570 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:56.553404 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e6f829a-4773-4740-a9e4-328091ee920f-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-pnbfj\" (UID: \"9e6f829a-4773-4740-a9e4-328091ee920f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" Nov 01 06:19:56.553570 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:56.553545 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl257\" (UniqueName: \"kubernetes.io/projected/9e6f829a-4773-4740-a9e4-328091ee920f-kube-api-access-tl257\") pod \"maas-default-gateway-openshift-default-855f9574cf-pnbfj\" (UID: \"9e6f829a-4773-4740-a9e4-328091ee920f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" Nov 01 06:19:56.553659 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:56.553586 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e6f829a-4773-4740-a9e4-328091ee920f-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-pnbfj\" (UID: \"9e6f829a-4773-4740-a9e4-328091ee920f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" Nov 01 06:19:56.575827 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:56.575790 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj"] Nov 01 06:19:56.655034 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:56.654986 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e6f829a-4773-4740-a9e4-328091ee920f-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-pnbfj\" (UID: \"9e6f829a-4773-4740-a9e4-328091ee920f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" Nov 01 06:19:56.655520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:56.655152 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e6f829a-4773-4740-a9e4-328091ee920f-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-pnbfj\" (UID: \"9e6f829a-4773-4740-a9e4-328091ee920f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" Nov 01 06:19:56.655520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:56.655196 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e6f829a-4773-4740-a9e4-328091ee920f-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-pnbfj\" (UID: \"9e6f829a-4773-4740-a9e4-328091ee920f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" Nov 01 06:19:56.655520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:56.655241 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e6f829a-4773-4740-a9e4-328091ee920f-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-pnbfj\" (UID: \"9e6f829a-4773-4740-a9e4-328091ee920f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" Nov 01 06:19:56.655520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:56.655312 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e6f829a-4773-4740-a9e4-328091ee920f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-pnbfj\" (UID: \"9e6f829a-4773-4740-a9e4-328091ee920f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" Nov 01 06:19:56.655520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:56.655348 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e6f829a-4773-4740-a9e4-328091ee920f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-pnbfj\" (UID: \"9e6f829a-4773-4740-a9e4-328091ee920f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" Nov 01 06:19:56.655520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:56.655379 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e6f829a-4773-4740-a9e4-328091ee920f-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-pnbfj\" (UID: \"9e6f829a-4773-4740-a9e4-328091ee920f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" Nov 01 06:19:56.655520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:56.655463 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e6f829a-4773-4740-a9e4-328091ee920f-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-pnbfj\" (UID: \"9e6f829a-4773-4740-a9e4-328091ee920f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" Nov 01 06:19:56.655520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:56.655518 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl257\" (UniqueName: \"kubernetes.io/projected/9e6f829a-4773-4740-a9e4-328091ee920f-kube-api-access-tl257\") pod \"maas-default-gateway-openshift-default-855f9574cf-pnbfj\" (UID: \"9e6f829a-4773-4740-a9e4-328091ee920f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" Nov 01 06:19:56.655944 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:56.655518 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e6f829a-4773-4740-a9e4-328091ee920f-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-pnbfj\" (UID: \"9e6f829a-4773-4740-a9e4-328091ee920f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" Nov 01 06:19:56.655944 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:56.655591 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e6f829a-4773-4740-a9e4-328091ee920f-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-pnbfj\" (UID: \"9e6f829a-4773-4740-a9e4-328091ee920f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" Nov 01 06:19:56.656030 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:56.655937 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e6f829a-4773-4740-a9e4-328091ee920f-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-pnbfj\" (UID: \"9e6f829a-4773-4740-a9e4-328091ee920f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" Nov 01 06:19:56.656079 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:56.656047 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e6f829a-4773-4740-a9e4-328091ee920f-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-pnbfj\" (UID: \"9e6f829a-4773-4740-a9e4-328091ee920f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" Nov 01 06:19:56.656286 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:56.656260 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e6f829a-4773-4740-a9e4-328091ee920f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-pnbfj\" (UID: \"9e6f829a-4773-4740-a9e4-328091ee920f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" Nov 01 06:19:56.657782 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:56.657751 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e6f829a-4773-4740-a9e4-328091ee920f-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-pnbfj\" (UID: \"9e6f829a-4773-4740-a9e4-328091ee920f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" Nov 01 06:19:56.658031 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:56.658011 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e6f829a-4773-4740-a9e4-328091ee920f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-pnbfj\" (UID: \"9e6f829a-4773-4740-a9e4-328091ee920f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" Nov 01 06:19:56.716827 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:56.716778 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e6f829a-4773-4740-a9e4-328091ee920f-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-pnbfj\" (UID: \"9e6f829a-4773-4740-a9e4-328091ee920f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" Nov 01 06:19:56.717070 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:56.716962 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl257\" (UniqueName: \"kubernetes.io/projected/9e6f829a-4773-4740-a9e4-328091ee920f-kube-api-access-tl257\") pod \"maas-default-gateway-openshift-default-855f9574cf-pnbfj\" (UID: \"9e6f829a-4773-4740-a9e4-328091ee920f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" Nov 01 06:19:56.785089 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:56.785050 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" Nov 01 06:19:57.220782 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:57.220744 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj"] Nov 01 06:19:57.221204 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:19:57.221176 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e6f829a_4773_4740_a9e4_328091ee920f.slice/crio-4849ae5aa5919e938b3490a481560d993b0eafa430e441ea9434f24bf45d78d9 WatchSource:0}: Error finding container 4849ae5aa5919e938b3490a481560d993b0eafa430e441ea9434f24bf45d78d9: Status 404 returned error can't find the container with id 4849ae5aa5919e938b3490a481560d993b0eafa430e441ea9434f24bf45d78d9 Nov 01 06:19:57.223846 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:57.223807 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:19:57.223946 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:57.223879 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:19:57.223946 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:57.223906 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:19:57.261630 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:57.261587 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" event={"ID":"9e6f829a-4773-4740-a9e4-328091ee920f","Type":"ContainerStarted","Data":"4849ae5aa5919e938b3490a481560d993b0eafa430e441ea9434f24bf45d78d9"} Nov 01 06:19:57.263557 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:57.263522 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-cd78f1/e2e-usage-sim-54291c-7c768bf864-pjszd" event={"ID":"6e0b1e48-0511-4ef7-8f73-1106983b5b33","Type":"ContainerStarted","Data":"dbf32ba40689e755ac7a70b3822931f131352e80384eb04c5f62191003d74273"} Nov 01 06:19:57.263698 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:57.263566 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-cd78f1/e2e-usage-sim-54291c-7c768bf864-pjszd" event={"ID":"6e0b1e48-0511-4ef7-8f73-1106983b5b33","Type":"ContainerStarted","Data":"d5c0b5566c523f1455424b834800a788c0790b9732ea7e694e2a548e33a8daf1"} Nov 01 06:19:57.263757 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:57.263708 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-cd78f1/e2e-usage-sim-54291c-7c768bf864-pjszd" Nov 01 06:19:57.268867 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:57.268842 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-cd78f1/e2e-usage-sim-54291c-7c768bf864-pjszd" Nov 01 06:19:57.382574 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:57.382511 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-cd78f1/e2e-usage-sim-54291c-7c768bf864-pjszd" podStartSLOduration=2.38249746 podStartE2EDuration="2.38249746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:19:57.379011148 +0000 UTC m=+2076.034128318" watchObservedRunningTime="2026-11-01 06:19:57.38249746 +0000 UTC m=+2076.037614640" Nov 01 06:19:57.669337 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:57.669302 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h"] Nov 01 06:19:58.271535 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:58.271448 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" event={"ID":"9e6f829a-4773-4740-a9e4-328091ee920f","Type":"ContainerStarted","Data":"ae843ba1e8e079961f95d588c37ea16c76aeb714ead9448c841153ddbb0f57c0"} Nov 01 06:19:58.415504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:58.394091 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" podStartSLOduration=2.394072366 podStartE2EDuration="2.394072366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:19:58.392663 +0000 UTC m=+2077.047780192" watchObservedRunningTime="2026-11-01 06:19:58.394072366 +0000 UTC m=+2077.049189533" Nov 01 06:19:58.785530 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:58.785472 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" Nov 01 06:19:58.787241 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:58.787214 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pnbfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 01 06:19:58.787373 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:58.787263 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" podUID="9e6f829a-4773-4740-a9e4-328091ee920f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 01 06:19:59.647949 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:59.647908 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm" podUID="9decea03-910e-4838-b4f0-b6065bf7cf8b" containerName="payload-processing" containerID="cri-o://527532ce574fabca966ed05cf4410ae3c90456b5afca18773e7e1fcc65329f0e" gracePeriod=25 Nov 01 06:19:59.786298 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:59.786263 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pnbfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 01 06:19:59.786719 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:59.786319 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" podUID="9e6f829a-4773-4740-a9e4-328091ee920f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 01 06:19:59.957046 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:19:59.957018 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm" Nov 01 06:20:00.022857 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:00.022816 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52" Nov 01 06:20:00.035884 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:00.035846 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52" Nov 01 06:20:00.092584 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:00.092546 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9decea03-910e-4838-b4f0-b6065bf7cf8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9decea03-910e-4838-b4f0-b6065bf7cf8b-plugins-config-volume\") pod \"9decea03-910e-4838-b4f0-b6065bf7cf8b\" (UID: \"9decea03-910e-4838-b4f0-b6065bf7cf8b\") " Nov 01 06:20:00.092793 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:00.092648 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9decea03-910e-4838-b4f0-b6065bf7cf8b-kube-api-access-2vzcv\" (UniqueName: \"kubernetes.io/projected/9decea03-910e-4838-b4f0-b6065bf7cf8b-kube-api-access-2vzcv\") pod \"9decea03-910e-4838-b4f0-b6065bf7cf8b\" (UID: \"9decea03-910e-4838-b4f0-b6065bf7cf8b\") " Nov 01 06:20:00.092793 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:00.092747 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9decea03-910e-4838-b4f0-b6065bf7cf8b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9decea03-910e-4838-b4f0-b6065bf7cf8b-routing-overlay-volume\") pod \"9decea03-910e-4838-b4f0-b6065bf7cf8b\" (UID: \"9decea03-910e-4838-b4f0-b6065bf7cf8b\") " Nov 01 06:20:00.092926 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:00.092806 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9decea03-910e-4838-b4f0-b6065bf7cf8b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9decea03-910e-4838-b4f0-b6065bf7cf8b-provider-credentials\") pod \"9decea03-910e-4838-b4f0-b6065bf7cf8b\" (UID: \"9decea03-910e-4838-b4f0-b6065bf7cf8b\") " Nov 01 06:20:00.093088 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:00.092962 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9decea03-910e-4838-b4f0-b6065bf7cf8b-plugins-config-volume" pod "9decea03-910e-4838-b4f0-b6065bf7cf8b" (UID: "9decea03-910e-4838-b4f0-b6065bf7cf8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:20:00.093160 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:00.093140 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9decea03-910e-4838-b4f0-b6065bf7cf8b-routing-overlay-volume" pod "9decea03-910e-4838-b4f0-b6065bf7cf8b" (UID: "9decea03-910e-4838-b4f0-b6065bf7cf8b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:20:00.097544 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:00.093522 5040 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9decea03-910e-4838-b4f0-b6065bf7cf8b-plugins-config-volume\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:20:00.097544 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:00.093551 5040 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9decea03-910e-4838-b4f0-b6065bf7cf8b-routing-overlay-volume\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:20:00.121505 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:00.120089 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9decea03-910e-4838-b4f0-b6065bf7cf8b-provider-credentials" pod "9decea03-910e-4838-b4f0-b6065bf7cf8b" (UID: "9decea03-910e-4838-b4f0-b6065bf7cf8b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:20:00.121505 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:00.120212 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9decea03-910e-4838-b4f0-b6065bf7cf8b-kube-api-access-2vzcv" pod "9decea03-910e-4838-b4f0-b6065bf7cf8b" (UID: "9decea03-910e-4838-b4f0-b6065bf7cf8b"). InnerVolumeSpecName "kube-api-access-2vzcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:20:00.218565 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:00.217220 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vzcv\" (UniqueName: \"kubernetes.io/projected/9decea03-910e-4838-b4f0-b6065bf7cf8b-kube-api-access-2vzcv\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:20:00.218565 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:00.217264 5040 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9decea03-910e-4838-b4f0-b6065bf7cf8b-provider-credentials\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:20:00.285841 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:00.285805 5040 generic.go:356] "Generic (PLEG): container finished" podID="9decea03-910e-4838-b4f0-b6065bf7cf8b" containerID="527532ce574fabca966ed05cf4410ae3c90456b5afca18773e7e1fcc65329f0e" exitCode=0 Nov 01 06:20:00.286052 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:00.285850 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm" Nov 01 06:20:00.286052 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:00.285889 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm" event={"ID":"9decea03-910e-4838-b4f0-b6065bf7cf8b","Type":"ContainerDied","Data":"527532ce574fabca966ed05cf4410ae3c90456b5afca18773e7e1fcc65329f0e"} Nov 01 06:20:00.286052 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:00.285951 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm" event={"ID":"9decea03-910e-4838-b4f0-b6065bf7cf8b","Type":"ContainerDied","Data":"efe7a1b349c12a4dd488924e5d2786697e66dac567c12cae363f31798be9daa6"} Nov 01 06:20:00.286052 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:00.285976 5040 scope.go:122] "RemoveContainer" containerID="527532ce574fabca966ed05cf4410ae3c90456b5afca18773e7e1fcc65329f0e" Nov 01 06:20:00.321679 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:00.321658 5040 scope.go:122] "RemoveContainer" containerID="527532ce574fabca966ed05cf4410ae3c90456b5afca18773e7e1fcc65329f0e" Nov 01 06:20:00.322005 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:20:00.321975 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"527532ce574fabca966ed05cf4410ae3c90456b5afca18773e7e1fcc65329f0e\": container with ID starting with 527532ce574fabca966ed05cf4410ae3c90456b5afca18773e7e1fcc65329f0e not found: ID does not exist" containerID="527532ce574fabca966ed05cf4410ae3c90456b5afca18773e7e1fcc65329f0e" Nov 01 06:20:00.322051 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:00.322019 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"527532ce574fabca966ed05cf4410ae3c90456b5afca18773e7e1fcc65329f0e"} err="failed to get container status \"527532ce574fabca966ed05cf4410ae3c90456b5afca18773e7e1fcc65329f0e\": rpc error: code = NotFound desc = could not find container \"527532ce574fabca966ed05cf4410ae3c90456b5afca18773e7e1fcc65329f0e\": container with ID starting with 527532ce574fabca966ed05cf4410ae3c90456b5afca18773e7e1fcc65329f0e not found: ID does not exist" Nov 01 06:20:00.325709 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:00.325686 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52" Nov 01 06:20:00.454175 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:00.454139 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm"] Nov 01 06:20:00.457520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:00.457463 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876722rm"] Nov 01 06:20:00.786468 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:00.786433 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pnbfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 01 06:20:00.786854 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:00.786514 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" podUID="9e6f829a-4773-4740-a9e4-328091ee920f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 01 06:20:01.536526 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:01.536453 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9decea03-910e-4838-b4f0-b6065bf7cf8b" path="/var/lib/kubelet/pods/9decea03-910e-4838-b4f0-b6065bf7cf8b/volumes" Nov 01 06:20:01.785861 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:01.785819 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pnbfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 01 06:20:01.786035 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:01.785893 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" podUID="9e6f829a-4773-4740-a9e4-328091ee920f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 01 06:20:02.156852 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:02.156811 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4cda2f-g526c"] Nov 01 06:20:02.157562 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:02.157538 5040 cpu_manager.go:414] "containerMap: removing container" podUID="9decea03-910e-4838-b4f0-b6065bf7cf8b" containerName="payload-processing" Nov 01 06:20:02.157622 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:02.157564 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9decea03-910e-4838-b4f0-b6065bf7cf8b" containerName="payload-processing" Nov 01 06:20:02.157758 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:02.157745 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="9decea03-910e-4838-b4f0-b6065bf7cf8b" containerName="payload-processing" Nov 01 06:20:02.163236 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:02.163212 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4cda2f-g526c" Nov 01 06:20:02.222509 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:02.221747 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:20:02.222509 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:02.221865 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:20:02.222509 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:02.221926 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:20:02.222509 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:02.221985 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:20:02.228499 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:02.228040 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4cda2f-g526c"] Nov 01 06:20:02.343862 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:02.343820 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4xll\" (UniqueName: \"kubernetes.io/projected/cc1be5f9-fc7d-405e-8774-466fbd320cc7-kube-api-access-x4xll\") pod \"maas-api-revoke-keys-e2e-ait-2a4cda2f-g526c\" (UID: \"cc1be5f9-fc7d-405e-8774-466fbd320cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4cda2f-g526c" Nov 01 06:20:02.344068 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:02.343976 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1be5f9-fc7d-405e-8774-466fbd320cc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a4cda2f-g526c\" (UID: \"cc1be5f9-fc7d-405e-8774-466fbd320cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4cda2f-g526c" Nov 01 06:20:02.344068 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:02.344018 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1be5f9-fc7d-405e-8774-466fbd320cc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a4cda2f-g526c\" (UID: \"cc1be5f9-fc7d-405e-8774-466fbd320cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4cda2f-g526c" Nov 01 06:20:02.447617 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:02.446137 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1be5f9-fc7d-405e-8774-466fbd320cc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a4cda2f-g526c\" (UID: \"cc1be5f9-fc7d-405e-8774-466fbd320cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4cda2f-g526c" Nov 01 06:20:02.447617 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:02.446203 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1be5f9-fc7d-405e-8774-466fbd320cc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a4cda2f-g526c\" (UID: \"cc1be5f9-fc7d-405e-8774-466fbd320cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4cda2f-g526c" Nov 01 06:20:02.447617 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:02.446249 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4xll\" (UniqueName: \"kubernetes.io/projected/cc1be5f9-fc7d-405e-8774-466fbd320cc7-kube-api-access-x4xll\") pod \"maas-api-revoke-keys-e2e-ait-2a4cda2f-g526c\" (UID: \"cc1be5f9-fc7d-405e-8774-466fbd320cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4cda2f-g526c" Nov 01 06:20:02.447929 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:02.447692 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1be5f9-fc7d-405e-8774-466fbd320cc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a4cda2f-g526c\" (UID: \"cc1be5f9-fc7d-405e-8774-466fbd320cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4cda2f-g526c" Nov 01 06:20:02.455198 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:02.455163 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1be5f9-fc7d-405e-8774-466fbd320cc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a4cda2f-g526c\" (UID: \"cc1be5f9-fc7d-405e-8774-466fbd320cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4cda2f-g526c" Nov 01 06:20:02.531961 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:02.531923 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4xll\" (UniqueName: \"kubernetes.io/projected/cc1be5f9-fc7d-405e-8774-466fbd320cc7-kube-api-access-x4xll\") pod \"maas-api-revoke-keys-e2e-ait-2a4cda2f-g526c\" (UID: \"cc1be5f9-fc7d-405e-8774-466fbd320cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4cda2f-g526c" Nov 01 06:20:02.670691 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:02.670633 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h" podUID="f6f444d8-62e1-4f3f-b7dd-9d31399d7534" containerName="payload-processing" containerID="cri-o://73aebf27dcbb750aa51cc323fd99999b834ae423e9746cddcf4c6c5bbfddb513" gracePeriod=25 Nov 01 06:20:02.779070 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:02.778818 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4cda2f-g526c" Nov 01 06:20:02.785979 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:02.785941 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pnbfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 01 06:20:02.786200 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:02.786153 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" podUID="9e6f829a-4773-4740-a9e4-328091ee920f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 01 06:20:03.234681 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:03.234601 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h" Nov 01 06:20:03.288657 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:03.288624 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4cda2f-g526c"] Nov 01 06:20:03.289291 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:20:03.289262 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc1be5f9_fc7d_405e_8774_466fbd320cc7.slice/crio-177f98bc88517645ff10083b53f64b774063a49910e46bc59c75e67af26f739d WatchSource:0}: Error finding container 177f98bc88517645ff10083b53f64b774063a49910e46bc59c75e67af26f739d: Status 404 returned error can't find the container with id 177f98bc88517645ff10083b53f64b774063a49910e46bc59c75e67af26f739d Nov 01 06:20:03.337580 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:03.337538 5040 generic.go:356] "Generic (PLEG): container finished" podID="f6f444d8-62e1-4f3f-b7dd-9d31399d7534" containerID="73aebf27dcbb750aa51cc323fd99999b834ae423e9746cddcf4c6c5bbfddb513" exitCode=0 Nov 01 06:20:03.337766 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:03.337658 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h" event={"ID":"f6f444d8-62e1-4f3f-b7dd-9d31399d7534","Type":"ContainerDied","Data":"73aebf27dcbb750aa51cc323fd99999b834ae423e9746cddcf4c6c5bbfddb513"} Nov 01 06:20:03.337766 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:03.337709 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h" event={"ID":"f6f444d8-62e1-4f3f-b7dd-9d31399d7534","Type":"ContainerDied","Data":"76e272a2afcd3216135aad9767299ef763bd010b9eb297c7095ed6cf36ed10aa"} Nov 01 06:20:03.337766 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:03.337739 5040 scope.go:122] "RemoveContainer" containerID="73aebf27dcbb750aa51cc323fd99999b834ae423e9746cddcf4c6c5bbfddb513" Nov 01 06:20:03.338003 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:03.337883 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h" Nov 01 06:20:03.340293 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:03.340265 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4cda2f-g526c" event={"ID":"cc1be5f9-fc7d-405e-8774-466fbd320cc7","Type":"ContainerStarted","Data":"177f98bc88517645ff10083b53f64b774063a49910e46bc59c75e67af26f739d"} Nov 01 06:20:03.352726 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:03.352703 5040 scope.go:122] "RemoveContainer" containerID="73aebf27dcbb750aa51cc323fd99999b834ae423e9746cddcf4c6c5bbfddb513" Nov 01 06:20:03.353121 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:20:03.353097 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73aebf27dcbb750aa51cc323fd99999b834ae423e9746cddcf4c6c5bbfddb513\": container with ID starting with 73aebf27dcbb750aa51cc323fd99999b834ae423e9746cddcf4c6c5bbfddb513 not found: ID does not exist" containerID="73aebf27dcbb750aa51cc323fd99999b834ae423e9746cddcf4c6c5bbfddb513" Nov 01 06:20:03.353189 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:03.353131 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73aebf27dcbb750aa51cc323fd99999b834ae423e9746cddcf4c6c5bbfddb513"} err="failed to get container status \"73aebf27dcbb750aa51cc323fd99999b834ae423e9746cddcf4c6c5bbfddb513\": rpc error: code = NotFound desc = could not find container \"73aebf27dcbb750aa51cc323fd99999b834ae423e9746cddcf4c6c5bbfddb513\": container with ID starting with 73aebf27dcbb750aa51cc323fd99999b834ae423e9746cddcf4c6c5bbfddb513 not found: ID does not exist" Nov 01 06:20:03.362112 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:03.361540 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6f444d8-62e1-4f3f-b7dd-9d31399d7534-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f6f444d8-62e1-4f3f-b7dd-9d31399d7534-provider-credentials\") pod \"f6f444d8-62e1-4f3f-b7dd-9d31399d7534\" (UID: \"f6f444d8-62e1-4f3f-b7dd-9d31399d7534\") " Nov 01 06:20:03.362112 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:03.361610 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6f444d8-62e1-4f3f-b7dd-9d31399d7534-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6f444d8-62e1-4f3f-b7dd-9d31399d7534-plugins-config-volume\") pod \"f6f444d8-62e1-4f3f-b7dd-9d31399d7534\" (UID: \"f6f444d8-62e1-4f3f-b7dd-9d31399d7534\") " Nov 01 06:20:03.362112 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:03.361745 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6f444d8-62e1-4f3f-b7dd-9d31399d7534-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f6f444d8-62e1-4f3f-b7dd-9d31399d7534-routing-overlay-volume\") pod \"f6f444d8-62e1-4f3f-b7dd-9d31399d7534\" (UID: \"f6f444d8-62e1-4f3f-b7dd-9d31399d7534\") " Nov 01 06:20:03.362112 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:03.361839 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6f444d8-62e1-4f3f-b7dd-9d31399d7534-kube-api-access-h6mnx\" (UniqueName: \"kubernetes.io/projected/f6f444d8-62e1-4f3f-b7dd-9d31399d7534-kube-api-access-h6mnx\") pod \"f6f444d8-62e1-4f3f-b7dd-9d31399d7534\" (UID: \"f6f444d8-62e1-4f3f-b7dd-9d31399d7534\") " Nov 01 06:20:03.362764 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:03.362287 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6f444d8-62e1-4f3f-b7dd-9d31399d7534-routing-overlay-volume" pod "f6f444d8-62e1-4f3f-b7dd-9d31399d7534" (UID: "f6f444d8-62e1-4f3f-b7dd-9d31399d7534"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:20:03.363310 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:03.363288 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6f444d8-62e1-4f3f-b7dd-9d31399d7534-plugins-config-volume" pod "f6f444d8-62e1-4f3f-b7dd-9d31399d7534" (UID: "f6f444d8-62e1-4f3f-b7dd-9d31399d7534"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:20:03.365225 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:03.365197 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6f444d8-62e1-4f3f-b7dd-9d31399d7534-provider-credentials" pod "f6f444d8-62e1-4f3f-b7dd-9d31399d7534" (UID: "f6f444d8-62e1-4f3f-b7dd-9d31399d7534"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:20:03.365987 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:03.365957 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6f444d8-62e1-4f3f-b7dd-9d31399d7534-kube-api-access-h6mnx" pod "f6f444d8-62e1-4f3f-b7dd-9d31399d7534" (UID: "f6f444d8-62e1-4f3f-b7dd-9d31399d7534"). InnerVolumeSpecName "kube-api-access-h6mnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:20:03.463160 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:03.463061 5040 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f6f444d8-62e1-4f3f-b7dd-9d31399d7534-provider-credentials\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:20:03.463160 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:03.463093 5040 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6f444d8-62e1-4f3f-b7dd-9d31399d7534-plugins-config-volume\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:20:03.463160 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:03.463114 5040 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f6f444d8-62e1-4f3f-b7dd-9d31399d7534-routing-overlay-volume\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:20:03.463160 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:03.463124 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6mnx\" (UniqueName: \"kubernetes.io/projected/f6f444d8-62e1-4f3f-b7dd-9d31399d7534-kube-api-access-h6mnx\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:20:03.762647 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:03.762601 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h"] Nov 01 06:20:03.771862 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:03.771824 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dz6r5h"] Nov 01 06:20:03.785804 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:03.785762 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pnbfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 01 06:20:03.785996 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:03.785822 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" podUID="9e6f829a-4773-4740-a9e4-328091ee920f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 01 06:20:04.356506 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:04.355931 5040 generic.go:356] "Generic (PLEG): container finished" podID="cc1be5f9-fc7d-405e-8774-466fbd320cc7" containerID="72c05979b7b2d6e3c17ee26d315c73f00ccb936b27af060b9403f95bcafeb80a" exitCode=0 Nov 01 06:20:04.356506 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:04.356023 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4cda2f-g526c" event={"ID":"cc1be5f9-fc7d-405e-8774-466fbd320cc7","Type":"ContainerDied","Data":"72c05979b7b2d6e3c17ee26d315c73f00ccb936b27af060b9403f95bcafeb80a"} Nov 01 06:20:04.785970 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:04.785930 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pnbfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 01 06:20:04.786137 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:04.785987 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" podUID="9e6f829a-4773-4740-a9e4-328091ee920f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 01 06:20:05.532129 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:05.530983 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6f444d8-62e1-4f3f-b7dd-9d31399d7534" path="/var/lib/kubelet/pods/f6f444d8-62e1-4f3f-b7dd-9d31399d7534/volumes" Nov 01 06:20:05.583516 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:05.583473 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4cda2f-g526c" Nov 01 06:20:05.710971 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:05.710875 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc1be5f9-fc7d-405e-8774-466fbd320cc7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1be5f9-fc7d-405e-8774-466fbd320cc7-maas-api-openshift-ca\") pod \"cc1be5f9-fc7d-405e-8774-466fbd320cc7\" (UID: \"cc1be5f9-fc7d-405e-8774-466fbd320cc7\") " Nov 01 06:20:05.711164 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:05.710977 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc1be5f9-fc7d-405e-8774-466fbd320cc7-kube-api-access-x4xll\" (UniqueName: \"kubernetes.io/projected/cc1be5f9-fc7d-405e-8774-466fbd320cc7-kube-api-access-x4xll\") pod \"cc1be5f9-fc7d-405e-8774-466fbd320cc7\" (UID: \"cc1be5f9-fc7d-405e-8774-466fbd320cc7\") " Nov 01 06:20:05.711164 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:05.711039 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc1be5f9-fc7d-405e-8774-466fbd320cc7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1be5f9-fc7d-405e-8774-466fbd320cc7-maas-api-xks-ca\") pod \"cc1be5f9-fc7d-405e-8774-466fbd320cc7\" (UID: \"cc1be5f9-fc7d-405e-8774-466fbd320cc7\") " Nov 01 06:20:05.712054 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:05.712021 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc1be5f9-fc7d-405e-8774-466fbd320cc7-maas-api-openshift-ca" pod "cc1be5f9-fc7d-405e-8774-466fbd320cc7" (UID: "cc1be5f9-fc7d-405e-8774-466fbd320cc7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:20:05.717988 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:05.717641 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc1be5f9-fc7d-405e-8774-466fbd320cc7-kube-api-access-x4xll" pod "cc1be5f9-fc7d-405e-8774-466fbd320cc7" (UID: "cc1be5f9-fc7d-405e-8774-466fbd320cc7"). InnerVolumeSpecName "kube-api-access-x4xll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:20:05.717988 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:05.717818 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc1be5f9-fc7d-405e-8774-466fbd320cc7-maas-api-xks-ca" pod "cc1be5f9-fc7d-405e-8774-466fbd320cc7" (UID: "cc1be5f9-fc7d-405e-8774-466fbd320cc7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:20:05.813115 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:05.813064 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1be5f9-fc7d-405e-8774-466fbd320cc7-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:20:05.813115 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:05.813112 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4xll\" (UniqueName: \"kubernetes.io/projected/cc1be5f9-fc7d-405e-8774-466fbd320cc7-kube-api-access-x4xll\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:20:05.813386 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:05.813129 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1be5f9-fc7d-405e-8774-466fbd320cc7-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:20:06.381505 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:06.381309 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" Nov 01 06:20:06.391502 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:06.382629 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" Nov 01 06:20:06.391502 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:06.387801 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" Nov 01 06:20:06.417503 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:06.414904 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4cda2f-g526c" event={"ID":"cc1be5f9-fc7d-405e-8774-466fbd320cc7","Type":"ContainerDied","Data":"177f98bc88517645ff10083b53f64b774063a49910e46bc59c75e67af26f739d"} Nov 01 06:20:06.417503 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:06.414969 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="177f98bc88517645ff10083b53f64b774063a49910e46bc59c75e67af26f739d" Nov 01 06:20:06.417503 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:06.415039 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4cda2f-g526c" Nov 01 06:20:09.974729 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:20:09.974686 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:20:11.590971 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:11.590930 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8"] Nov 01 06:20:11.591646 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:11.591625 5040 cpu_manager.go:414] "containerMap: removing container" podUID="cc1be5f9-fc7d-405e-8774-466fbd320cc7" containerName="revoke-keys" Nov 01 06:20:11.591735 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:11.591648 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc1be5f9-fc7d-405e-8774-466fbd320cc7" containerName="revoke-keys" Nov 01 06:20:11.591735 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:11.591669 5040 cpu_manager.go:414] "containerMap: removing container" podUID="f6f444d8-62e1-4f3f-b7dd-9d31399d7534" containerName="payload-processing" Nov 01 06:20:11.591735 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:11.591676 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6f444d8-62e1-4f3f-b7dd-9d31399d7534" containerName="payload-processing" Nov 01 06:20:11.591904 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:11.591851 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc1be5f9-fc7d-405e-8774-466fbd320cc7" containerName="revoke-keys" Nov 01 06:20:11.591904 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:11.591866 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6f444d8-62e1-4f3f-b7dd-9d31399d7534" containerName="payload-processing" Nov 01 06:20:11.595695 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:11.595671 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" Nov 01 06:20:11.698512 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:11.698441 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6760e57-3c38-4dda-84e0-75b99b3a70a6-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-jxfb8\" (UID: \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" Nov 01 06:20:11.698786 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:11.698527 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6760e57-3c38-4dda-84e0-75b99b3a70a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-jxfb8\" (UID: \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" Nov 01 06:20:11.698786 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:11.698631 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6760e57-3c38-4dda-84e0-75b99b3a70a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-jxfb8\" (UID: \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" Nov 01 06:20:11.698786 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:11.698743 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6760e57-3c38-4dda-84e0-75b99b3a70a6-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-jxfb8\" (UID: \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" Nov 01 06:20:11.698907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:11.698814 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6760e57-3c38-4dda-84e0-75b99b3a70a6-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-jxfb8\" (UID: \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" Nov 01 06:20:11.698907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:11.698850 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6760e57-3c38-4dda-84e0-75b99b3a70a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-jxfb8\" (UID: \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" Nov 01 06:20:11.699019 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:11.698972 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6760e57-3c38-4dda-84e0-75b99b3a70a6-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-jxfb8\" (UID: \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" Nov 01 06:20:11.699075 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:11.699017 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6760e57-3c38-4dda-84e0-75b99b3a70a6-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-jxfb8\" (UID: \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" Nov 01 06:20:11.699169 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:11.699152 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtz6r\" (UniqueName: \"kubernetes.io/projected/b6760e57-3c38-4dda-84e0-75b99b3a70a6-kube-api-access-jtz6r\") pod \"maas-default-gateway-openshift-default-855f9574cf-jxfb8\" (UID: \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" Nov 01 06:20:11.704441 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:11.704409 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8"] Nov 01 06:20:11.800554 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:11.800517 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6760e57-3c38-4dda-84e0-75b99b3a70a6-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-jxfb8\" (UID: \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" Nov 01 06:20:11.800751 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:11.800567 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6760e57-3c38-4dda-84e0-75b99b3a70a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-jxfb8\" (UID: \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" Nov 01 06:20:11.800751 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:11.800615 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6760e57-3c38-4dda-84e0-75b99b3a70a6-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-jxfb8\" (UID: \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" Nov 01 06:20:11.800751 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:11.800646 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6760e57-3c38-4dda-84e0-75b99b3a70a6-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-jxfb8\" (UID: \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" Nov 01 06:20:11.800751 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:11.800731 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtz6r\" (UniqueName: \"kubernetes.io/projected/b6760e57-3c38-4dda-84e0-75b99b3a70a6-kube-api-access-jtz6r\") pod \"maas-default-gateway-openshift-default-855f9574cf-jxfb8\" (UID: \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" Nov 01 06:20:11.800980 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:11.800768 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6760e57-3c38-4dda-84e0-75b99b3a70a6-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-jxfb8\" (UID: \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" Nov 01 06:20:11.800980 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:11.800826 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6760e57-3c38-4dda-84e0-75b99b3a70a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-jxfb8\" (UID: \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" Nov 01 06:20:11.800980 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:11.800861 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6760e57-3c38-4dda-84e0-75b99b3a70a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-jxfb8\" (UID: \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" Nov 01 06:20:11.800980 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:11.800912 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6760e57-3c38-4dda-84e0-75b99b3a70a6-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-jxfb8\" (UID: \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" Nov 01 06:20:11.801186 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:11.800977 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6760e57-3c38-4dda-84e0-75b99b3a70a6-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-jxfb8\" (UID: \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" Nov 01 06:20:11.801186 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:11.801075 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6760e57-3c38-4dda-84e0-75b99b3a70a6-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-jxfb8\" (UID: \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" Nov 01 06:20:11.801297 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:11.801219 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6760e57-3c38-4dda-84e0-75b99b3a70a6-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-jxfb8\" (UID: \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" Nov 01 06:20:11.801360 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:11.801325 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6760e57-3c38-4dda-84e0-75b99b3a70a6-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-jxfb8\" (UID: \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" Nov 01 06:20:11.801423 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:11.801404 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6760e57-3c38-4dda-84e0-75b99b3a70a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-jxfb8\" (UID: \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" Nov 01 06:20:11.803213 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:11.803191 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6760e57-3c38-4dda-84e0-75b99b3a70a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-jxfb8\" (UID: \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" Nov 01 06:20:11.803358 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:11.803342 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6760e57-3c38-4dda-84e0-75b99b3a70a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-jxfb8\" (UID: \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" Nov 01 06:20:11.862410 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:11.862333 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6760e57-3c38-4dda-84e0-75b99b3a70a6-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-jxfb8\" (UID: \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" Nov 01 06:20:11.862583 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:11.862407 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtz6r\" (UniqueName: \"kubernetes.io/projected/b6760e57-3c38-4dda-84e0-75b99b3a70a6-kube-api-access-jtz6r\") pod \"maas-default-gateway-openshift-default-855f9574cf-jxfb8\" (UID: \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" Nov 01 06:20:11.909383 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:11.909340 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" Nov 01 06:20:12.343499 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:12.343446 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8"] Nov 01 06:20:12.343855 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:20:12.343830 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6760e57_3c38_4dda_84e0_75b99b3a70a6.slice/crio-99ef2e27d9e89140578cbe09df8098905f85fcfd9d0f4dfc7c5b4ecc81399a3e WatchSource:0}: Error finding container 99ef2e27d9e89140578cbe09df8098905f85fcfd9d0f4dfc7c5b4ecc81399a3e: Status 404 returned error can't find the container with id 99ef2e27d9e89140578cbe09df8098905f85fcfd9d0f4dfc7c5b4ecc81399a3e Nov 01 06:20:12.346373 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:12.346340 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:20:12.346439 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:12.346406 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:20:12.346503 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:12.346436 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:20:12.465261 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:12.465226 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" event={"ID":"b6760e57-3c38-4dda-84e0-75b99b3a70a6","Type":"ContainerStarted","Data":"4804258cb3e72067ce2e5d304ec340f0c2267f9beef737db9b108dbe00fa85ec"} Nov 01 06:20:12.465443 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:12.465277 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" event={"ID":"b6760e57-3c38-4dda-84e0-75b99b3a70a6","Type":"ContainerStarted","Data":"99ef2e27d9e89140578cbe09df8098905f85fcfd9d0f4dfc7c5b4ecc81399a3e"} Nov 01 06:20:12.591545 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:12.591406 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" podStartSLOduration=1.591389094 podStartE2EDuration="1.591389094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:20:12.587416795 +0000 UTC m=+2091.242533969" watchObservedRunningTime="2026-11-01 06:20:12.591389094 +0000 UTC m=+2091.246506253" Nov 01 06:20:12.911577 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:12.911458 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" Nov 01 06:20:13.913503 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:13.912914 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-jxfb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 01 06:20:13.913503 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:13.912995 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" podUID="b6760e57-3c38-4dda-84e0-75b99b3a70a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 01 06:20:14.911502 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:14.910943 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-jxfb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 01 06:20:14.911502 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:14.911016 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" podUID="b6760e57-3c38-4dda-84e0-75b99b3a70a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 01 06:20:15.914525 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:15.912932 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-jxfb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 01 06:20:15.914525 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:15.913019 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" podUID="b6760e57-3c38-4dda-84e0-75b99b3a70a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 01 06:20:16.913503 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:16.913428 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-jxfb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 01 06:20:16.913748 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:16.913539 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" podUID="b6760e57-3c38-4dda-84e0-75b99b3a70a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 01 06:20:17.916504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:17.914993 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-jxfb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 01 06:20:17.916504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:17.915073 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" podUID="b6760e57-3c38-4dda-84e0-75b99b3a70a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 01 06:20:18.910418 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:18.910371 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-jxfb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 01 06:20:18.910634 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:18.910461 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" podUID="b6760e57-3c38-4dda-84e0-75b99b3a70a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 01 06:20:19.910305 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:19.910270 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-jxfb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 01 06:20:19.910709 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:19.910325 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" podUID="b6760e57-3c38-4dda-84e0-75b99b3a70a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 01 06:20:20.915064 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:20.915034 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" Nov 01 06:20:20.915569 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:20.915511 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" Nov 01 06:20:20.916384 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:20.916362 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" Nov 01 06:20:21.842105 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:21.842076 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 06:20:21.845074 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:21.845043 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 06:20:23.047816 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:23.047779 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e004500a-tmq5k"] Nov 01 06:20:23.098080 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:23.098042 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e004500a-tmq5k"] Nov 01 06:20:23.515605 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:23.515572 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89cda59d-10d7-434f-890e-6e5c5b1caf45" path="/var/lib/kubelet/pods/89cda59d-10d7-434f-890e-6e5c5b1caf45/volumes" Nov 01 06:20:33.048248 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:33.048157 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77d25194-klk5n"] Nov 01 06:20:33.098553 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:33.098519 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77d25194-klk5n"] Nov 01 06:20:33.515207 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:33.515172 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a810b4d-d7e5-4d1b-8299-71fae1652240" path="/var/lib/kubelet/pods/7a810b4d-d7e5-4d1b-8299-71fae1652240/volumes" Nov 01 06:20:39.973731 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:20:39.973685 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:20:47.356572 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:47.356536 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4cda2f-g526c"] Nov 01 06:20:47.406793 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:47.406759 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4cda2f-g526c"] Nov 01 06:20:47.525450 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:47.525415 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc1be5f9-fc7d-405e-8774-466fbd320cc7" path="/var/lib/kubelet/pods/cc1be5f9-fc7d-405e-8774-466fbd320cc7/volumes" Nov 01 06:20:47.557866 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:47.557830 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4"] Nov 01 06:20:47.562263 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:47.562239 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4" Nov 01 06:20:47.617706 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:47.616726 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-99e79c\"/\"openshift-service-ca.crt\"" Nov 01 06:20:47.617706 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:47.616935 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-99e79c\"/\"kube-root-ca.crt\"" Nov 01 06:20:47.620169 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:47.620142 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4"] Nov 01 06:20:47.666536 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:47.666500 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-99e79c\"/\"auth-test-model-99e79c-kserve-self-signed-certs\"" Nov 01 06:20:47.666713 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:47.666650 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-99e79c\"/\"default-dockercfg-bmpgc\"" Nov 01 06:20:47.710811 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:47.710764 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plhb7\" (UniqueName: \"kubernetes.io/projected/3c295272-f68b-4568-8735-99890a86dd44-kube-api-access-plhb7\") pod \"auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4\" (UID: \"3c295272-f68b-4568-8735-99890a86dd44\") " pod="ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4" Nov 01 06:20:47.711062 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:47.710881 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c295272-f68b-4568-8735-99890a86dd44-dshm\") pod \"auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4\" (UID: \"3c295272-f68b-4568-8735-99890a86dd44\") " pod="ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4" Nov 01 06:20:47.711062 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:47.710912 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c295272-f68b-4568-8735-99890a86dd44-model-cache\") pod \"auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4\" (UID: \"3c295272-f68b-4568-8735-99890a86dd44\") " pod="ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4" Nov 01 06:20:47.711062 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:47.710944 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c295272-f68b-4568-8735-99890a86dd44-tmp-dir\") pod \"auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4\" (UID: \"3c295272-f68b-4568-8735-99890a86dd44\") " pod="ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4" Nov 01 06:20:47.711062 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:47.710988 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c295272-f68b-4568-8735-99890a86dd44-tls-certs\") pod \"auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4\" (UID: \"3c295272-f68b-4568-8735-99890a86dd44\") " pod="ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4" Nov 01 06:20:47.711062 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:47.711025 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c295272-f68b-4568-8735-99890a86dd44-home\") pod \"auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4\" (UID: \"3c295272-f68b-4568-8735-99890a86dd44\") " pod="ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4" Nov 01 06:20:47.812573 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:47.812531 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plhb7\" (UniqueName: \"kubernetes.io/projected/3c295272-f68b-4568-8735-99890a86dd44-kube-api-access-plhb7\") pod \"auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4\" (UID: \"3c295272-f68b-4568-8735-99890a86dd44\") " pod="ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4" Nov 01 06:20:47.812767 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:47.812643 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c295272-f68b-4568-8735-99890a86dd44-dshm\") pod \"auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4\" (UID: \"3c295272-f68b-4568-8735-99890a86dd44\") " pod="ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4" Nov 01 06:20:47.812767 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:47.812671 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c295272-f68b-4568-8735-99890a86dd44-model-cache\") pod \"auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4\" (UID: \"3c295272-f68b-4568-8735-99890a86dd44\") " pod="ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4" Nov 01 06:20:47.812767 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:47.812702 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c295272-f68b-4568-8735-99890a86dd44-tmp-dir\") pod \"auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4\" (UID: \"3c295272-f68b-4568-8735-99890a86dd44\") " pod="ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4" Nov 01 06:20:47.812767 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:47.812753 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c295272-f68b-4568-8735-99890a86dd44-tls-certs\") pod \"auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4\" (UID: \"3c295272-f68b-4568-8735-99890a86dd44\") " pod="ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4" Nov 01 06:20:47.812970 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:47.812790 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c295272-f68b-4568-8735-99890a86dd44-home\") pod \"auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4\" (UID: \"3c295272-f68b-4568-8735-99890a86dd44\") " pod="ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4" Nov 01 06:20:47.813178 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:47.813147 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c295272-f68b-4568-8735-99890a86dd44-model-cache\") pod \"auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4\" (UID: \"3c295272-f68b-4568-8735-99890a86dd44\") " pod="ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4" Nov 01 06:20:47.813324 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:47.813180 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c295272-f68b-4568-8735-99890a86dd44-tmp-dir\") pod \"auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4\" (UID: \"3c295272-f68b-4568-8735-99890a86dd44\") " pod="ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4" Nov 01 06:20:47.813324 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:47.813226 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c295272-f68b-4568-8735-99890a86dd44-home\") pod \"auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4\" (UID: \"3c295272-f68b-4568-8735-99890a86dd44\") " pod="ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4" Nov 01 06:20:47.814984 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:47.814963 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c295272-f68b-4568-8735-99890a86dd44-dshm\") pod \"auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4\" (UID: \"3c295272-f68b-4568-8735-99890a86dd44\") " pod="ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4" Nov 01 06:20:47.815268 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:47.815251 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c295272-f68b-4568-8735-99890a86dd44-tls-certs\") pod \"auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4\" (UID: \"3c295272-f68b-4568-8735-99890a86dd44\") " pod="ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4" Nov 01 06:20:47.874266 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:47.874189 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plhb7\" (UniqueName: \"kubernetes.io/projected/3c295272-f68b-4568-8735-99890a86dd44-kube-api-access-plhb7\") pod \"auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4\" (UID: \"3c295272-f68b-4568-8735-99890a86dd44\") " pod="ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4" Nov 01 06:20:47.874409 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:47.874393 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4" Nov 01 06:20:48.316813 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:48.316770 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4"] Nov 01 06:20:48.319942 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:20:48.319898 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c295272_f68b_4568_8735_99890a86dd44.slice/crio-c37940da6cb819d3925c9156f69746a16058ecee76e22a29476647d9a56aa292 WatchSource:0}: Error finding container c37940da6cb819d3925c9156f69746a16058ecee76e22a29476647d9a56aa292: Status 404 returned error can't find the container with id c37940da6cb819d3925c9156f69746a16058ecee76e22a29476647d9a56aa292 Nov 01 06:20:48.710240 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:48.710135 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4" event={"ID":"3c295272-f68b-4568-8735-99890a86dd44","Type":"ContainerStarted","Data":"d4ed5231833bef0218ce370bf31c4646860854fa1161b92e400824e0d71430f7"} Nov 01 06:20:48.710240 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:48.710187 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4" event={"ID":"3c295272-f68b-4568-8735-99890a86dd44","Type":"ContainerStarted","Data":"c37940da6cb819d3925c9156f69746a16058ecee76e22a29476647d9a56aa292"} Nov 01 06:20:48.828104 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:48.828052 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4" podStartSLOduration=1.828038867 podStartE2EDuration="1.828038867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:20:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:20:48.827183149 +0000 UTC m=+2127.482300317" watchObservedRunningTime="2026-11-01 06:20:48.828038867 +0000 UTC m=+2127.483156034" Nov 01 06:20:48.874842 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:48.874808 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4" Nov 01 06:20:49.057149 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:49.054544 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33b1bc69-tljnk"] Nov 01 06:20:49.059632 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:49.059600 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33b1bc69-tljnk"] Nov 01 06:20:49.517695 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:49.517653 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="709afc4b-6023-48bc-a35b-20f4ba689011" path="/var/lib/kubelet/pods/709afc4b-6023-48bc-a35b-20f4ba689011/volumes" Nov 01 06:20:57.875620 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:57.875581 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4" Nov 01 06:20:57.893883 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:57.893850 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4" Nov 01 06:20:58.794890 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:20:58.794854 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4" Nov 01 06:21:09.974308 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:21:09.974272 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:21:11.454186 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:11.454152 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh"] Nov 01 06:21:11.454720 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:11.454417 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" podUID="dab6153a-193e-49d3-b9a4-3aee41f22bf1" containerName="istio-proxy" containerID="cri-o://74badabff435274324d172c70b2728a5fc70138b18215b7af7cc5daf3d11fc12" gracePeriod=30 Nov 01 06:21:11.499706 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:11.499670 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95"] Nov 01 06:21:11.499968 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:11.499918 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" podUID="78ec0c34-0221-488f-a075-7f05da1461e8" containerName="istio-proxy" containerID="cri-o://368e949ce48e01f0526a701b6c69b201b9f8d84fc33868256bec025a8d11bc4b" gracePeriod=30 Nov 01 06:21:16.763757 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.763728 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" Nov 01 06:21:16.801704 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.801674 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" Nov 01 06:21:16.837163 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.837128 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dab6153a-193e-49d3-b9a4-3aee41f22bf1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dab6153a-193e-49d3-b9a4-3aee41f22bf1-istio-podinfo\") pod \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\" (UID: \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\") " Nov 01 06:21:16.837323 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.837170 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dab6153a-193e-49d3-b9a4-3aee41f22bf1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab6153a-193e-49d3-b9a4-3aee41f22bf1-credential-socket\") pod \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\" (UID: \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\") " Nov 01 06:21:16.837323 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.837199 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dab6153a-193e-49d3-b9a4-3aee41f22bf1-kube-api-access-mqcjc\" (UniqueName: \"kubernetes.io/projected/dab6153a-193e-49d3-b9a4-3aee41f22bf1-kube-api-access-mqcjc\") pod \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\" (UID: \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\") " Nov 01 06:21:16.837323 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.837231 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dab6153a-193e-49d3-b9a4-3aee41f22bf1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dab6153a-193e-49d3-b9a4-3aee41f22bf1-istio-data\") pod \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\" (UID: \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\") " Nov 01 06:21:16.837323 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.837310 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dab6153a-193e-49d3-b9a4-3aee41f22bf1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dab6153a-193e-49d3-b9a4-3aee41f22bf1-workload-certs\") pod \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\" (UID: \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\") " Nov 01 06:21:16.837589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.837362 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dab6153a-193e-49d3-b9a4-3aee41f22bf1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dab6153a-193e-49d3-b9a4-3aee41f22bf1-istiod-ca-cert\") pod \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\" (UID: \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\") " Nov 01 06:21:16.837589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.837493 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dab6153a-193e-49d3-b9a4-3aee41f22bf1-istio-token\" (UniqueName: \"kubernetes.io/projected/dab6153a-193e-49d3-b9a4-3aee41f22bf1-istio-token\") pod \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\" (UID: \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\") " Nov 01 06:21:16.837589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.837521 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dab6153a-193e-49d3-b9a4-3aee41f22bf1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dab6153a-193e-49d3-b9a4-3aee41f22bf1-istio-envoy\") pod \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\" (UID: \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\") " Nov 01 06:21:16.837589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.837525 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab6153a-193e-49d3-b9a4-3aee41f22bf1-credential-socket" pod "dab6153a-193e-49d3-b9a4-3aee41f22bf1" (UID: "dab6153a-193e-49d3-b9a4-3aee41f22bf1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:21:16.837589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.837565 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dab6153a-193e-49d3-b9a4-3aee41f22bf1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab6153a-193e-49d3-b9a4-3aee41f22bf1-workload-socket\") pod \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\" (UID: \"dab6153a-193e-49d3-b9a4-3aee41f22bf1\") " Nov 01 06:21:16.837853 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.837606 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab6153a-193e-49d3-b9a4-3aee41f22bf1-istio-data" pod "dab6153a-193e-49d3-b9a4-3aee41f22bf1" (UID: "dab6153a-193e-49d3-b9a4-3aee41f22bf1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:21:16.837853 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.837617 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab6153a-193e-49d3-b9a4-3aee41f22bf1-workload-certs" pod "dab6153a-193e-49d3-b9a4-3aee41f22bf1" (UID: "dab6153a-193e-49d3-b9a4-3aee41f22bf1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:21:16.837853 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.837762 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab6153a-193e-49d3-b9a4-3aee41f22bf1-workload-socket" pod "dab6153a-193e-49d3-b9a4-3aee41f22bf1" (UID: "dab6153a-193e-49d3-b9a4-3aee41f22bf1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:21:16.838018 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.837941 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dab6153a-193e-49d3-b9a4-3aee41f22bf1-istiod-ca-cert" pod "dab6153a-193e-49d3-b9a4-3aee41f22bf1" (UID: "dab6153a-193e-49d3-b9a4-3aee41f22bf1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:21:16.838123 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.838099 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab6153a-193e-49d3-b9a4-3aee41f22bf1-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:21:16.838206 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.838124 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dab6153a-193e-49d3-b9a4-3aee41f22bf1-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:21:16.838206 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.838138 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dab6153a-193e-49d3-b9a4-3aee41f22bf1-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:21:16.838206 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.838151 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dab6153a-193e-49d3-b9a4-3aee41f22bf1-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:21:16.838206 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.838165 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab6153a-193e-49d3-b9a4-3aee41f22bf1-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:21:16.839678 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.839642 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dab6153a-193e-49d3-b9a4-3aee41f22bf1-istio-token" pod "dab6153a-193e-49d3-b9a4-3aee41f22bf1" (UID: "dab6153a-193e-49d3-b9a4-3aee41f22bf1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:21:16.839790 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.839754 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dab6153a-193e-49d3-b9a4-3aee41f22bf1-kube-api-access-mqcjc" pod "dab6153a-193e-49d3-b9a4-3aee41f22bf1" (UID: "dab6153a-193e-49d3-b9a4-3aee41f22bf1"). InnerVolumeSpecName "kube-api-access-mqcjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:21:16.839873 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.839797 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dab6153a-193e-49d3-b9a4-3aee41f22bf1-istio-podinfo" pod "dab6153a-193e-49d3-b9a4-3aee41f22bf1" (UID: "dab6153a-193e-49d3-b9a4-3aee41f22bf1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:21:16.839967 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.839945 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab6153a-193e-49d3-b9a4-3aee41f22bf1-istio-envoy" pod "dab6153a-193e-49d3-b9a4-3aee41f22bf1" (UID: "dab6153a-193e-49d3-b9a4-3aee41f22bf1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:21:16.900923 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.900886 5040 generic.go:356] "Generic (PLEG): container finished" podID="dab6153a-193e-49d3-b9a4-3aee41f22bf1" containerID="74badabff435274324d172c70b2728a5fc70138b18215b7af7cc5daf3d11fc12" exitCode=0 Nov 01 06:21:16.901078 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.901068 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" Nov 01 06:21:16.901146 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.901083 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" event={"ID":"dab6153a-193e-49d3-b9a4-3aee41f22bf1","Type":"ContainerDied","Data":"74badabff435274324d172c70b2728a5fc70138b18215b7af7cc5daf3d11fc12"} Nov 01 06:21:16.901146 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.901141 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh" event={"ID":"dab6153a-193e-49d3-b9a4-3aee41f22bf1","Type":"ContainerDied","Data":"2473dcb18c9a54f7d0375f33994d7827254e40f623ab55b12a024c904200b817"} Nov 01 06:21:16.901232 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.901170 5040 scope.go:122] "RemoveContainer" containerID="74badabff435274324d172c70b2728a5fc70138b18215b7af7cc5daf3d11fc12" Nov 01 06:21:16.903088 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.903062 5040 generic.go:356] "Generic (PLEG): container finished" podID="78ec0c34-0221-488f-a075-7f05da1461e8" containerID="368e949ce48e01f0526a701b6c69b201b9f8d84fc33868256bec025a8d11bc4b" exitCode=0 Nov 01 06:21:16.903210 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.903126 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" Nov 01 06:21:16.903210 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.903137 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" event={"ID":"78ec0c34-0221-488f-a075-7f05da1461e8","Type":"ContainerDied","Data":"368e949ce48e01f0526a701b6c69b201b9f8d84fc33868256bec025a8d11bc4b"} Nov 01 06:21:16.903210 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.903179 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95" event={"ID":"78ec0c34-0221-488f-a075-7f05da1461e8","Type":"ContainerDied","Data":"004697bf623c39b879aa28a393e582eb472984cf513d296a729f7622e47a978f"} Nov 01 06:21:16.914156 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.914133 5040 scope.go:122] "RemoveContainer" containerID="74badabff435274324d172c70b2728a5fc70138b18215b7af7cc5daf3d11fc12" Nov 01 06:21:16.914474 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:21:16.914446 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74badabff435274324d172c70b2728a5fc70138b18215b7af7cc5daf3d11fc12\": container with ID starting with 74badabff435274324d172c70b2728a5fc70138b18215b7af7cc5daf3d11fc12 not found: ID does not exist" containerID="74badabff435274324d172c70b2728a5fc70138b18215b7af7cc5daf3d11fc12" Nov 01 06:21:16.914557 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.914514 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74badabff435274324d172c70b2728a5fc70138b18215b7af7cc5daf3d11fc12"} err="failed to get container status \"74badabff435274324d172c70b2728a5fc70138b18215b7af7cc5daf3d11fc12\": rpc error: code = NotFound desc = could not find container \"74badabff435274324d172c70b2728a5fc70138b18215b7af7cc5daf3d11fc12\": container with ID starting with 74badabff435274324d172c70b2728a5fc70138b18215b7af7cc5daf3d11fc12 not found: ID does not exist" Nov 01 06:21:16.914557 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.914532 5040 scope.go:122] "RemoveContainer" containerID="368e949ce48e01f0526a701b6c69b201b9f8d84fc33868256bec025a8d11bc4b" Nov 01 06:21:16.925008 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.924926 5040 scope.go:122] "RemoveContainer" containerID="368e949ce48e01f0526a701b6c69b201b9f8d84fc33868256bec025a8d11bc4b" Nov 01 06:21:16.926963 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:21:16.926938 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"368e949ce48e01f0526a701b6c69b201b9f8d84fc33868256bec025a8d11bc4b\": container with ID starting with 368e949ce48e01f0526a701b6c69b201b9f8d84fc33868256bec025a8d11bc4b not found: ID does not exist" containerID="368e949ce48e01f0526a701b6c69b201b9f8d84fc33868256bec025a8d11bc4b" Nov 01 06:21:16.927052 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.926973 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"368e949ce48e01f0526a701b6c69b201b9f8d84fc33868256bec025a8d11bc4b"} err="failed to get container status \"368e949ce48e01f0526a701b6c69b201b9f8d84fc33868256bec025a8d11bc4b\": rpc error: code = NotFound desc = could not find container \"368e949ce48e01f0526a701b6c69b201b9f8d84fc33868256bec025a8d11bc4b\": container with ID starting with 368e949ce48e01f0526a701b6c69b201b9f8d84fc33868256bec025a8d11bc4b not found: ID does not exist" Nov 01 06:21:16.939077 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.939053 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78ec0c34-0221-488f-a075-7f05da1461e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78ec0c34-0221-488f-a075-7f05da1461e8-istio-podinfo\") pod \"78ec0c34-0221-488f-a075-7f05da1461e8\" (UID: \"78ec0c34-0221-488f-a075-7f05da1461e8\") " Nov 01 06:21:16.939198 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.939103 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78ec0c34-0221-488f-a075-7f05da1461e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78ec0c34-0221-488f-a075-7f05da1461e8-istio-envoy\") pod \"78ec0c34-0221-488f-a075-7f05da1461e8\" (UID: \"78ec0c34-0221-488f-a075-7f05da1461e8\") " Nov 01 06:21:16.939269 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.939231 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78ec0c34-0221-488f-a075-7f05da1461e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ec0c34-0221-488f-a075-7f05da1461e8-workload-socket\") pod \"78ec0c34-0221-488f-a075-7f05da1461e8\" (UID: \"78ec0c34-0221-488f-a075-7f05da1461e8\") " Nov 01 06:21:16.939331 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.939279 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78ec0c34-0221-488f-a075-7f05da1461e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78ec0c34-0221-488f-a075-7f05da1461e8-istio-data\") pod \"78ec0c34-0221-488f-a075-7f05da1461e8\" (UID: \"78ec0c34-0221-488f-a075-7f05da1461e8\") " Nov 01 06:21:16.939331 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.939326 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78ec0c34-0221-488f-a075-7f05da1461e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ec0c34-0221-488f-a075-7f05da1461e8-credential-socket\") pod \"78ec0c34-0221-488f-a075-7f05da1461e8\" (UID: \"78ec0c34-0221-488f-a075-7f05da1461e8\") " Nov 01 06:21:16.939440 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.939369 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78ec0c34-0221-488f-a075-7f05da1461e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78ec0c34-0221-488f-a075-7f05da1461e8-workload-certs\") pod \"78ec0c34-0221-488f-a075-7f05da1461e8\" (UID: \"78ec0c34-0221-488f-a075-7f05da1461e8\") " Nov 01 06:21:16.939526 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.939440 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78ec0c34-0221-488f-a075-7f05da1461e8-istio-token\" (UniqueName: \"kubernetes.io/projected/78ec0c34-0221-488f-a075-7f05da1461e8-istio-token\") pod \"78ec0c34-0221-488f-a075-7f05da1461e8\" (UID: \"78ec0c34-0221-488f-a075-7f05da1461e8\") " Nov 01 06:21:16.939526 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.939519 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78ec0c34-0221-488f-a075-7f05da1461e8-kube-api-access-2q4w6\" (UniqueName: \"kubernetes.io/projected/78ec0c34-0221-488f-a075-7f05da1461e8-kube-api-access-2q4w6\") pod \"78ec0c34-0221-488f-a075-7f05da1461e8\" (UID: \"78ec0c34-0221-488f-a075-7f05da1461e8\") " Nov 01 06:21:16.939650 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.939595 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78ec0c34-0221-488f-a075-7f05da1461e8-istio-data" pod "78ec0c34-0221-488f-a075-7f05da1461e8" (UID: "78ec0c34-0221-488f-a075-7f05da1461e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:21:16.940327 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.939766 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78ec0c34-0221-488f-a075-7f05da1461e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78ec0c34-0221-488f-a075-7f05da1461e8-istiod-ca-cert\") pod \"78ec0c34-0221-488f-a075-7f05da1461e8\" (UID: \"78ec0c34-0221-488f-a075-7f05da1461e8\") " Nov 01 06:21:16.940327 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.939780 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78ec0c34-0221-488f-a075-7f05da1461e8-workload-certs" pod "78ec0c34-0221-488f-a075-7f05da1461e8" (UID: "78ec0c34-0221-488f-a075-7f05da1461e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:21:16.940327 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.940082 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78ec0c34-0221-488f-a075-7f05da1461e8-istiod-ca-cert" pod "78ec0c34-0221-488f-a075-7f05da1461e8" (UID: "78ec0c34-0221-488f-a075-7f05da1461e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:21:16.940327 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.940210 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78ec0c34-0221-488f-a075-7f05da1461e8-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:21:16.940327 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.940241 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78ec0c34-0221-488f-a075-7f05da1461e8-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:21:16.940327 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.940256 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dab6153a-193e-49d3-b9a4-3aee41f22bf1-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:21:16.940327 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.940268 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dab6153a-193e-49d3-b9a4-3aee41f22bf1-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:21:16.940327 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.940280 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78ec0c34-0221-488f-a075-7f05da1461e8-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:21:16.940327 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.940276 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78ec0c34-0221-488f-a075-7f05da1461e8-workload-socket" pod "78ec0c34-0221-488f-a075-7f05da1461e8" (UID: "78ec0c34-0221-488f-a075-7f05da1461e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:21:16.940327 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.940292 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dab6153a-193e-49d3-b9a4-3aee41f22bf1-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:21:16.940327 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.940304 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqcjc\" (UniqueName: \"kubernetes.io/projected/dab6153a-193e-49d3-b9a4-3aee41f22bf1-kube-api-access-mqcjc\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:21:16.941178 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.941145 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78ec0c34-0221-488f-a075-7f05da1461e8-credential-socket" pod "78ec0c34-0221-488f-a075-7f05da1461e8" (UID: "78ec0c34-0221-488f-a075-7f05da1461e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:21:16.941646 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.941624 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78ec0c34-0221-488f-a075-7f05da1461e8-istio-envoy" pod "78ec0c34-0221-488f-a075-7f05da1461e8" (UID: "78ec0c34-0221-488f-a075-7f05da1461e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:21:16.941963 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.941945 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78ec0c34-0221-488f-a075-7f05da1461e8-kube-api-access-2q4w6" pod "78ec0c34-0221-488f-a075-7f05da1461e8" (UID: "78ec0c34-0221-488f-a075-7f05da1461e8"). InnerVolumeSpecName "kube-api-access-2q4w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:21:16.942368 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.942344 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78ec0c34-0221-488f-a075-7f05da1461e8-istio-podinfo" pod "78ec0c34-0221-488f-a075-7f05da1461e8" (UID: "78ec0c34-0221-488f-a075-7f05da1461e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:21:16.942806 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:16.942785 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78ec0c34-0221-488f-a075-7f05da1461e8-istio-token" pod "78ec0c34-0221-488f-a075-7f05da1461e8" (UID: "78ec0c34-0221-488f-a075-7f05da1461e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:21:17.041491 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:17.041456 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78ec0c34-0221-488f-a075-7f05da1461e8-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:21:17.041674 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:17.041509 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78ec0c34-0221-488f-a075-7f05da1461e8-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:21:17.041674 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:17.041521 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ec0c34-0221-488f-a075-7f05da1461e8-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:21:17.041674 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:17.041530 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ec0c34-0221-488f-a075-7f05da1461e8-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:21:17.041674 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:17.041539 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78ec0c34-0221-488f-a075-7f05da1461e8-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:21:17.041674 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:17.041548 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q4w6\" (UniqueName: \"kubernetes.io/projected/78ec0c34-0221-488f-a075-7f05da1461e8-kube-api-access-2q4w6\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:21:17.051669 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:17.051628 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh"] Nov 01 06:21:17.057638 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:17.057602 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7vtxh"] Nov 01 06:21:17.326996 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:17.326914 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95"] Nov 01 06:21:17.331130 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:17.331101 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-jjs95"] Nov 01 06:21:17.515218 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:17.515184 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78ec0c34-0221-488f-a075-7f05da1461e8" path="/var/lib/kubelet/pods/78ec0c34-0221-488f-a075-7f05da1461e8/volumes" Nov 01 06:21:17.515686 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:17.515670 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dab6153a-193e-49d3-b9a4-3aee41f22bf1" path="/var/lib/kubelet/pods/dab6153a-193e-49d3-b9a4-3aee41f22bf1/volumes" Nov 01 06:21:20.920429 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:20.920394 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c93ac60-n7l49"] Nov 01 06:21:20.921102 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:20.921080 5040 cpu_manager.go:414] "containerMap: removing container" podUID="dab6153a-193e-49d3-b9a4-3aee41f22bf1" containerName="istio-proxy" Nov 01 06:21:20.921208 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:20.921116 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dab6153a-193e-49d3-b9a4-3aee41f22bf1" containerName="istio-proxy" Nov 01 06:21:20.921208 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:20.921154 5040 cpu_manager.go:414] "containerMap: removing container" podUID="78ec0c34-0221-488f-a075-7f05da1461e8" containerName="istio-proxy" Nov 01 06:21:20.921208 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:20.921163 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78ec0c34-0221-488f-a075-7f05da1461e8" containerName="istio-proxy" Nov 01 06:21:20.921377 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:20.921324 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="78ec0c34-0221-488f-a075-7f05da1461e8" containerName="istio-proxy" Nov 01 06:21:20.921377 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:20.921351 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="dab6153a-193e-49d3-b9a4-3aee41f22bf1" containerName="istio-proxy" Nov 01 06:21:20.925934 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:20.925909 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c93ac60-n7l49" Nov 01 06:21:20.981693 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:20.981658 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c93ac60-n7l49"] Nov 01 06:21:20.987353 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:20.987321 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a8fc69b-5644-4cf5-930a-785de80cbc46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c93ac60-n7l49\" (UID: \"5a8fc69b-5644-4cf5-930a-785de80cbc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c93ac60-n7l49" Nov 01 06:21:20.987496 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:20.987430 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a8fc69b-5644-4cf5-930a-785de80cbc46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c93ac60-n7l49\" (UID: \"5a8fc69b-5644-4cf5-930a-785de80cbc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c93ac60-n7l49" Nov 01 06:21:20.987559 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:20.987542 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qbk9\" (UniqueName: \"kubernetes.io/projected/5a8fc69b-5644-4cf5-930a-785de80cbc46-kube-api-access-9qbk9\") pod \"maas-api-revoke-sub-7c93ac60-n7l49\" (UID: \"5a8fc69b-5644-4cf5-930a-785de80cbc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c93ac60-n7l49" Nov 01 06:21:21.028417 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:21.028377 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:21:21.028603 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:21.028530 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:21:21.028603 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:21.028533 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:21:21.029435 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:21.029415 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:21:21.088614 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:21.088583 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a8fc69b-5644-4cf5-930a-785de80cbc46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c93ac60-n7l49\" (UID: \"5a8fc69b-5644-4cf5-930a-785de80cbc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c93ac60-n7l49" Nov 01 06:21:21.088758 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:21.088662 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a8fc69b-5644-4cf5-930a-785de80cbc46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c93ac60-n7l49\" (UID: \"5a8fc69b-5644-4cf5-930a-785de80cbc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c93ac60-n7l49" Nov 01 06:21:21.088758 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:21.088746 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qbk9\" (UniqueName: \"kubernetes.io/projected/5a8fc69b-5644-4cf5-930a-785de80cbc46-kube-api-access-9qbk9\") pod \"maas-api-revoke-sub-7c93ac60-n7l49\" (UID: \"5a8fc69b-5644-4cf5-930a-785de80cbc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c93ac60-n7l49" Nov 01 06:21:21.089454 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:21.089430 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a8fc69b-5644-4cf5-930a-785de80cbc46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c93ac60-n7l49\" (UID: \"5a8fc69b-5644-4cf5-930a-785de80cbc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c93ac60-n7l49" Nov 01 06:21:21.090928 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:21.090909 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a8fc69b-5644-4cf5-930a-785de80cbc46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c93ac60-n7l49\" (UID: \"5a8fc69b-5644-4cf5-930a-785de80cbc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c93ac60-n7l49" Nov 01 06:21:21.150726 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:21.150688 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qbk9\" (UniqueName: \"kubernetes.io/projected/5a8fc69b-5644-4cf5-930a-785de80cbc46-kube-api-access-9qbk9\") pod \"maas-api-revoke-sub-7c93ac60-n7l49\" (UID: \"5a8fc69b-5644-4cf5-930a-785de80cbc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c93ac60-n7l49" Nov 01 06:21:21.239674 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:21.239586 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c93ac60-n7l49" Nov 01 06:21:21.617718 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:21.617685 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c93ac60-n7l49"] Nov 01 06:21:21.618019 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:21:21.617995 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a8fc69b_5644_4cf5_930a_785de80cbc46.slice/crio-d5f86671d6c14d05fdc768dc3ae7c270e5131eddc0c4dba05a1dc8e4b47b761e WatchSource:0}: Error finding container d5f86671d6c14d05fdc768dc3ae7c270e5131eddc0c4dba05a1dc8e4b47b761e: Status 404 returned error can't find the container with id d5f86671d6c14d05fdc768dc3ae7c270e5131eddc0c4dba05a1dc8e4b47b761e Nov 01 06:21:21.938723 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:21.938688 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c93ac60-n7l49" event={"ID":"5a8fc69b-5644-4cf5-930a-785de80cbc46","Type":"ContainerStarted","Data":"08b3e565b35a53448663ddbec4921c43132e5c4284371251526167d488751a58"} Nov 01 06:21:21.939085 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:21.938773 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c93ac60-n7l49" event={"ID":"5a8fc69b-5644-4cf5-930a-785de80cbc46","Type":"ContainerStarted","Data":"d5f86671d6c14d05fdc768dc3ae7c270e5131eddc0c4dba05a1dc8e4b47b761e"} Nov 01 06:21:22.057621 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:22.057562 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c93ac60-n7l49" podStartSLOduration=2.057547794 podStartE2EDuration="2.057547794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:21:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:21:22.055286375 +0000 UTC m=+2160.710403543" watchObservedRunningTime="2026-11-01 06:21:22.057547794 +0000 UTC m=+2160.712664958" Nov 01 06:21:22.681132 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:22.681099 5040 scope.go:122] "RemoveContainer" containerID="78410d69807f3ef368d852ededfaf9e9b2938416c3ec90ab897d3a13448bc7ae" Nov 01 06:21:22.690629 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:22.690598 5040 scope.go:122] "RemoveContainer" containerID="fe03153a8809029929e8ff8888746579328f5fd1907317aeae7b8c4f17646fa0" Nov 01 06:21:22.699994 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:22.699970 5040 scope.go:122] "RemoveContainer" containerID="c3ca4dd27f8aaad821fb721c3f50f220a815c4288ac91f72a94ceb219b1d74c4" Nov 01 06:21:22.947357 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:22.947323 5040 generic.go:356] "Generic (PLEG): container finished" podID="5a8fc69b-5644-4cf5-930a-785de80cbc46" containerID="08b3e565b35a53448663ddbec4921c43132e5c4284371251526167d488751a58" exitCode=0 Nov 01 06:21:22.947809 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:22.947405 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c93ac60-n7l49" event={"ID":"5a8fc69b-5644-4cf5-930a-785de80cbc46","Type":"ContainerDied","Data":"08b3e565b35a53448663ddbec4921c43132e5c4284371251526167d488751a58"} Nov 01 06:21:24.089252 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:24.089223 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c93ac60-n7l49" Nov 01 06:21:24.220974 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:24.220897 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a8fc69b-5644-4cf5-930a-785de80cbc46-kube-api-access-9qbk9\" (UniqueName: \"kubernetes.io/projected/5a8fc69b-5644-4cf5-930a-785de80cbc46-kube-api-access-9qbk9\") pod \"5a8fc69b-5644-4cf5-930a-785de80cbc46\" (UID: \"5a8fc69b-5644-4cf5-930a-785de80cbc46\") " Nov 01 06:21:24.220974 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:24.220942 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a8fc69b-5644-4cf5-930a-785de80cbc46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a8fc69b-5644-4cf5-930a-785de80cbc46-maas-api-xks-ca\") pod \"5a8fc69b-5644-4cf5-930a-785de80cbc46\" (UID: \"5a8fc69b-5644-4cf5-930a-785de80cbc46\") " Nov 01 06:21:24.221166 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:24.221013 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a8fc69b-5644-4cf5-930a-785de80cbc46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a8fc69b-5644-4cf5-930a-785de80cbc46-maas-api-openshift-ca\") pod \"5a8fc69b-5644-4cf5-930a-785de80cbc46\" (UID: \"5a8fc69b-5644-4cf5-930a-785de80cbc46\") " Nov 01 06:21:24.221548 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:24.221514 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a8fc69b-5644-4cf5-930a-785de80cbc46-maas-api-openshift-ca" pod "5a8fc69b-5644-4cf5-930a-785de80cbc46" (UID: "5a8fc69b-5644-4cf5-930a-785de80cbc46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:21:24.223135 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:24.223105 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a8fc69b-5644-4cf5-930a-785de80cbc46-maas-api-xks-ca" pod "5a8fc69b-5644-4cf5-930a-785de80cbc46" (UID: "5a8fc69b-5644-4cf5-930a-785de80cbc46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:21:24.223250 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:24.223165 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a8fc69b-5644-4cf5-930a-785de80cbc46-kube-api-access-9qbk9" pod "5a8fc69b-5644-4cf5-930a-785de80cbc46" (UID: "5a8fc69b-5644-4cf5-930a-785de80cbc46"). InnerVolumeSpecName "kube-api-access-9qbk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:21:24.322454 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:24.322421 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qbk9\" (UniqueName: \"kubernetes.io/projected/5a8fc69b-5644-4cf5-930a-785de80cbc46-kube-api-access-9qbk9\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:21:24.322454 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:24.322449 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a8fc69b-5644-4cf5-930a-785de80cbc46-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:21:24.322454 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:24.322460 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a8fc69b-5644-4cf5-930a-785de80cbc46-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:21:24.968000 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:24.967966 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c93ac60-n7l49" Nov 01 06:21:24.968000 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:24.967995 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c93ac60-n7l49" event={"ID":"5a8fc69b-5644-4cf5-930a-785de80cbc46","Type":"ContainerDied","Data":"d5f86671d6c14d05fdc768dc3ae7c270e5131eddc0c4dba05a1dc8e4b47b761e"} Nov 01 06:21:24.968220 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:21:24.968063 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5f86671d6c14d05fdc768dc3ae7c270e5131eddc0c4dba05a1dc8e4b47b761e" Nov 01 06:21:39.982385 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:21:39.982343 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:22:09.972651 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:22:09.972616 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:22:12.530463 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:12.530420 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x"] Nov 01 06:22:12.531108 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:12.531086 5040 cpu_manager.go:414] "containerMap: removing container" podUID="5a8fc69b-5644-4cf5-930a-785de80cbc46" containerName="revoke-keys" Nov 01 06:22:12.531212 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:12.531110 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a8fc69b-5644-4cf5-930a-785de80cbc46" containerName="revoke-keys" Nov 01 06:22:12.531303 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:12.531289 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a8fc69b-5644-4cf5-930a-785de80cbc46" containerName="revoke-keys" Nov 01 06:22:12.534730 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:12.534708 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" Nov 01 06:22:12.573728 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:12.573694 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-8th8x\" (UID: \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" Nov 01 06:22:12.573844 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:12.573744 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-8th8x\" (UID: \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" Nov 01 06:22:12.573844 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:12.573820 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-8th8x\" (UID: \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" Nov 01 06:22:12.573922 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:12.573844 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-8th8x\" (UID: \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" Nov 01 06:22:12.573922 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:12.573863 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d9xk\" (UniqueName: \"kubernetes.io/projected/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-kube-api-access-6d9xk\") pod \"maas-default-gateway-openshift-default-855f9574cf-8th8x\" (UID: \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" Nov 01 06:22:12.574035 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:12.574013 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-8th8x\" (UID: \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" Nov 01 06:22:12.574073 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:12.574063 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-8th8x\" (UID: \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" Nov 01 06:22:12.574159 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:12.574139 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-8th8x\" (UID: \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" Nov 01 06:22:12.574195 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:12.574183 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-8th8x\" (UID: \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" Nov 01 06:22:12.641567 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:12.641537 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x"] Nov 01 06:22:12.674784 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:12.674759 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-8th8x\" (UID: \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" Nov 01 06:22:12.674935 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:12.674807 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-8th8x\" (UID: \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" Nov 01 06:22:12.674935 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:12.674838 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-8th8x\" (UID: \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" Nov 01 06:22:12.674935 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:12.674878 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-8th8x\" (UID: \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" Nov 01 06:22:12.674935 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:12.674930 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-8th8x\" (UID: \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" Nov 01 06:22:12.675158 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:12.674969 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-8th8x\" (UID: \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" Nov 01 06:22:12.675158 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:12.675012 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-8th8x\" (UID: \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" Nov 01 06:22:12.675158 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:12.675046 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-8th8x\" (UID: \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" Nov 01 06:22:12.675158 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:12.675073 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d9xk\" (UniqueName: \"kubernetes.io/projected/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-kube-api-access-6d9xk\") pod \"maas-default-gateway-openshift-default-855f9574cf-8th8x\" (UID: \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" Nov 01 06:22:12.675375 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:12.675203 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-8th8x\" (UID: \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" Nov 01 06:22:12.675449 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:12.675425 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-8th8x\" (UID: \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" Nov 01 06:22:12.675582 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:12.675559 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-8th8x\" (UID: \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" Nov 01 06:22:12.675643 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:12.675596 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-8th8x\" (UID: \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" Nov 01 06:22:12.675900 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:12.675881 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-8th8x\" (UID: \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" Nov 01 06:22:12.677763 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:12.677735 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-8th8x\" (UID: \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" Nov 01 06:22:12.677885 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:12.677868 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-8th8x\" (UID: \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" Nov 01 06:22:12.738494 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:12.738448 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d9xk\" (UniqueName: \"kubernetes.io/projected/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-kube-api-access-6d9xk\") pod \"maas-default-gateway-openshift-default-855f9574cf-8th8x\" (UID: \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" Nov 01 06:22:12.738638 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:12.738457 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-8th8x\" (UID: \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" Nov 01 06:22:12.849115 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:12.849036 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" Nov 01 06:22:13.276605 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:13.276577 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x"] Nov 01 06:22:13.277565 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:22:13.277534 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f4f1372_1a38_4d0b_b73b_9a9fa8b881de.slice/crio-869ac66553d4d9269e9ce25d3a01bf560a07cfe4790d757b867905bcd43255f5 WatchSource:0}: Error finding container 869ac66553d4d9269e9ce25d3a01bf560a07cfe4790d757b867905bcd43255f5: Status 404 returned error can't find the container with id 869ac66553d4d9269e9ce25d3a01bf560a07cfe4790d757b867905bcd43255f5 Nov 01 06:22:13.279969 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:13.279931 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:22:13.280074 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:13.280001 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:22:13.280074 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:13.280031 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:22:13.320274 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:13.320245 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" event={"ID":"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de","Type":"ContainerStarted","Data":"869ac66553d4d9269e9ce25d3a01bf560a07cfe4790d757b867905bcd43255f5"} Nov 01 06:22:14.328464 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:14.328431 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" event={"ID":"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de","Type":"ContainerStarted","Data":"20708662581ab371ed8b56bb6d07bff07a8f8d2a5dcf7ef260ea6f26dfb987e3"} Nov 01 06:22:14.551920 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:14.551865 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" podStartSLOduration=2.551851143 podStartE2EDuration="2.551851143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:22:14.499949541 +0000 UTC m=+2213.155066709" watchObservedRunningTime="2026-11-01 06:22:14.551851143 +0000 UTC m=+2213.206968384" Nov 01 06:22:14.849594 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:14.849557 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" Nov 01 06:22:14.851136 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:14.851106 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-8th8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 01 06:22:14.851277 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:14.851159 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" podUID="0f4f1372-1a38-4d0b-b73b-9a9fa8b881de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 01 06:22:15.850372 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:15.850339 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-8th8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 01 06:22:15.850813 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:15.850390 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" podUID="0f4f1372-1a38-4d0b-b73b-9a9fa8b881de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 01 06:22:16.850504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:16.850427 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-8th8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 01 06:22:16.850504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:16.850496 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" podUID="0f4f1372-1a38-4d0b-b73b-9a9fa8b881de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 01 06:22:17.850345 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:17.850295 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-8th8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 01 06:22:17.850578 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:17.850358 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" podUID="0f4f1372-1a38-4d0b-b73b-9a9fa8b881de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 01 06:22:18.849813 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:18.849775 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-8th8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 01 06:22:18.849994 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:18.849838 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" podUID="0f4f1372-1a38-4d0b-b73b-9a9fa8b881de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 01 06:22:19.850038 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:19.850002 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-8th8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 01 06:22:19.850432 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:19.850055 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" podUID="0f4f1372-1a38-4d0b-b73b-9a9fa8b881de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 01 06:22:20.850332 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:20.850289 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-8th8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 01 06:22:20.850767 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:20.850364 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" podUID="0f4f1372-1a38-4d0b-b73b-9a9fa8b881de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 01 06:22:21.988078 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:21.988048 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" Nov 01 06:22:21.988622 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:21.988494 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" Nov 01 06:22:21.989344 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:21.989311 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" Nov 01 06:22:22.743588 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:22.743549 5040 scope.go:122] "RemoveContainer" containerID="3524b61296cf9f90bcd66b681339fb4186f974a79909f766bac23349f3a76787" Nov 01 06:22:22.753441 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:22.753417 5040 scope.go:122] "RemoveContainer" containerID="1ea9a5207294cefd558fa2313f9e14a41261002dbc9488e8ee85fd71f9ff7a20" Nov 01 06:22:22.762428 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:22.762397 5040 scope.go:122] "RemoveContainer" containerID="0c543f791003f05efb5160c4f3d724c3b4b7da2f471195b1e0ad4000afb63bf3" Nov 01 06:22:39.972453 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:22:39.972413 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:22:49.721517 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:49.721465 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5d7100d-5pxzp"] Nov 01 06:22:49.725092 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:49.725075 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5d7100d-5pxzp" Nov 01 06:22:49.828854 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:49.828821 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:22:49.829119 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:49.829100 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:22:49.829171 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:49.829128 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:22:49.829171 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:49.829144 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:22:49.832164 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:49.832135 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5d7100d-5pxzp"] Nov 01 06:22:49.881985 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:49.881950 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbqkx\" (UniqueName: \"kubernetes.io/projected/492a4b92-30d2-438f-b1c4-a7f7c46c56dd-kube-api-access-xbqkx\") pod \"maas-api-revoke-keys-e2e-migrate-e5d7100d-5pxzp\" (UID: \"492a4b92-30d2-438f-b1c4-a7f7c46c56dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5d7100d-5pxzp" Nov 01 06:22:49.882141 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:49.881993 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492a4b92-30d2-438f-b1c4-a7f7c46c56dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e5d7100d-5pxzp\" (UID: \"492a4b92-30d2-438f-b1c4-a7f7c46c56dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5d7100d-5pxzp" Nov 01 06:22:49.882141 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:49.882131 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492a4b92-30d2-438f-b1c4-a7f7c46c56dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e5d7100d-5pxzp\" (UID: \"492a4b92-30d2-438f-b1c4-a7f7c46c56dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5d7100d-5pxzp" Nov 01 06:22:49.982945 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:49.982852 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492a4b92-30d2-438f-b1c4-a7f7c46c56dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e5d7100d-5pxzp\" (UID: \"492a4b92-30d2-438f-b1c4-a7f7c46c56dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5d7100d-5pxzp" Nov 01 06:22:49.983123 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:49.982968 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbqkx\" (UniqueName: \"kubernetes.io/projected/492a4b92-30d2-438f-b1c4-a7f7c46c56dd-kube-api-access-xbqkx\") pod \"maas-api-revoke-keys-e2e-migrate-e5d7100d-5pxzp\" (UID: \"492a4b92-30d2-438f-b1c4-a7f7c46c56dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5d7100d-5pxzp" Nov 01 06:22:49.983123 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:49.983002 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492a4b92-30d2-438f-b1c4-a7f7c46c56dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e5d7100d-5pxzp\" (UID: \"492a4b92-30d2-438f-b1c4-a7f7c46c56dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5d7100d-5pxzp" Nov 01 06:22:49.983709 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:49.983688 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492a4b92-30d2-438f-b1c4-a7f7c46c56dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e5d7100d-5pxzp\" (UID: \"492a4b92-30d2-438f-b1c4-a7f7c46c56dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5d7100d-5pxzp" Nov 01 06:22:49.985283 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:49.985264 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492a4b92-30d2-438f-b1c4-a7f7c46c56dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e5d7100d-5pxzp\" (UID: \"492a4b92-30d2-438f-b1c4-a7f7c46c56dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5d7100d-5pxzp" Nov 01 06:22:50.044675 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:50.044641 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbqkx\" (UniqueName: \"kubernetes.io/projected/492a4b92-30d2-438f-b1c4-a7f7c46c56dd-kube-api-access-xbqkx\") pod \"maas-api-revoke-keys-e2e-migrate-e5d7100d-5pxzp\" (UID: \"492a4b92-30d2-438f-b1c4-a7f7c46c56dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5d7100d-5pxzp" Nov 01 06:22:50.338162 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:50.338129 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5d7100d-5pxzp" Nov 01 06:22:50.715549 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:50.715516 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5d7100d-5pxzp"] Nov 01 06:22:50.716184 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:22:50.716157 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod492a4b92_30d2_438f_b1c4_a7f7c46c56dd.slice/crio-6b20db04b5eca78242bd1b3db120c514918b0c7e43a6311d9bb109dcaf9e53c7 WatchSource:0}: Error finding container 6b20db04b5eca78242bd1b3db120c514918b0c7e43a6311d9bb109dcaf9e53c7: Status 404 returned error can't find the container with id 6b20db04b5eca78242bd1b3db120c514918b0c7e43a6311d9bb109dcaf9e53c7 Nov 01 06:22:51.554738 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:51.554701 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5d7100d-5pxzp" event={"ID":"492a4b92-30d2-438f-b1c4-a7f7c46c56dd","Type":"ContainerStarted","Data":"943f772bb1d77d54cbd0730408d82de75a8ef3b63a9eaaf23221a9e01495fc9f"} Nov 01 06:22:51.555219 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:51.554751 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5d7100d-5pxzp" event={"ID":"492a4b92-30d2-438f-b1c4-a7f7c46c56dd","Type":"ContainerStarted","Data":"6b20db04b5eca78242bd1b3db120c514918b0c7e43a6311d9bb109dcaf9e53c7"} Nov 01 06:22:51.572854 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:22:51.572827 5040 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod492a4b92_30d2_438f_b1c4_a7f7c46c56dd.slice/crio-943f772bb1d77d54cbd0730408d82de75a8ef3b63a9eaaf23221a9e01495fc9f.scope\": RecentStats: unable to find data in memory cache]" Nov 01 06:22:51.673616 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:51.673551 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5d7100d-5pxzp" podStartSLOduration=2.673531715 podStartE2EDuration="2.673531715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:22:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:22:51.672806793 +0000 UTC m=+2250.327923961" watchObservedRunningTime="2026-11-01 06:22:51.673531715 +0000 UTC m=+2250.328648883" Nov 01 06:22:52.561984 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:52.561948 5040 generic.go:356] "Generic (PLEG): container finished" podID="492a4b92-30d2-438f-b1c4-a7f7c46c56dd" containerID="943f772bb1d77d54cbd0730408d82de75a8ef3b63a9eaaf23221a9e01495fc9f" exitCode=0 Nov 01 06:22:52.562403 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:52.562011 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5d7100d-5pxzp" event={"ID":"492a4b92-30d2-438f-b1c4-a7f7c46c56dd","Type":"ContainerDied","Data":"943f772bb1d77d54cbd0730408d82de75a8ef3b63a9eaaf23221a9e01495fc9f"} Nov 01 06:22:53.716069 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:53.716045 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5d7100d-5pxzp" Nov 01 06:22:53.721131 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:53.721111 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/492a4b92-30d2-438f-b1c4-a7f7c46c56dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492a4b92-30d2-438f-b1c4-a7f7c46c56dd-maas-api-openshift-ca\") pod \"492a4b92-30d2-438f-b1c4-a7f7c46c56dd\" (UID: \"492a4b92-30d2-438f-b1c4-a7f7c46c56dd\") " Nov 01 06:22:53.721243 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:53.721203 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/492a4b92-30d2-438f-b1c4-a7f7c46c56dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492a4b92-30d2-438f-b1c4-a7f7c46c56dd-maas-api-xks-ca\") pod \"492a4b92-30d2-438f-b1c4-a7f7c46c56dd\" (UID: \"492a4b92-30d2-438f-b1c4-a7f7c46c56dd\") " Nov 01 06:22:53.721317 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:53.721301 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/492a4b92-30d2-438f-b1c4-a7f7c46c56dd-kube-api-access-xbqkx\" (UniqueName: \"kubernetes.io/projected/492a4b92-30d2-438f-b1c4-a7f7c46c56dd-kube-api-access-xbqkx\") pod \"492a4b92-30d2-438f-b1c4-a7f7c46c56dd\" (UID: \"492a4b92-30d2-438f-b1c4-a7f7c46c56dd\") " Nov 01 06:22:53.721447 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:53.721425 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/492a4b92-30d2-438f-b1c4-a7f7c46c56dd-maas-api-openshift-ca" pod "492a4b92-30d2-438f-b1c4-a7f7c46c56dd" (UID: "492a4b92-30d2-438f-b1c4-a7f7c46c56dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:22:53.721700 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:53.721679 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492a4b92-30d2-438f-b1c4-a7f7c46c56dd-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:22:53.723173 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:53.723152 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/492a4b92-30d2-438f-b1c4-a7f7c46c56dd-maas-api-xks-ca" pod "492a4b92-30d2-438f-b1c4-a7f7c46c56dd" (UID: "492a4b92-30d2-438f-b1c4-a7f7c46c56dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:22:53.723258 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:53.723223 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492a4b92-30d2-438f-b1c4-a7f7c46c56dd-kube-api-access-xbqkx" pod "492a4b92-30d2-438f-b1c4-a7f7c46c56dd" (UID: "492a4b92-30d2-438f-b1c4-a7f7c46c56dd"). InnerVolumeSpecName "kube-api-access-xbqkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:22:53.822790 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:53.822702 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492a4b92-30d2-438f-b1c4-a7f7c46c56dd-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:22:53.822790 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:53.822733 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbqkx\" (UniqueName: \"kubernetes.io/projected/492a4b92-30d2-438f-b1c4-a7f7c46c56dd-kube-api-access-xbqkx\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:22:54.575548 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:54.575524 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5d7100d-5pxzp" Nov 01 06:22:54.575548 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:54.575539 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5d7100d-5pxzp" event={"ID":"492a4b92-30d2-438f-b1c4-a7f7c46c56dd","Type":"ContainerDied","Data":"6b20db04b5eca78242bd1b3db120c514918b0c7e43a6311d9bb109dcaf9e53c7"} Nov 01 06:22:54.575781 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:22:54.575588 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b20db04b5eca78242bd1b3db120c514918b0c7e43a6311d9bb109dcaf9e53c7" Nov 01 06:23:02.792049 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:02.792016 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx"] Nov 01 06:23:02.792587 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:02.792552 5040 cpu_manager.go:414] "containerMap: removing container" podUID="492a4b92-30d2-438f-b1c4-a7f7c46c56dd" containerName="revoke-keys" Nov 01 06:23:02.792587 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:02.792568 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="492a4b92-30d2-438f-b1c4-a7f7c46c56dd" containerName="revoke-keys" Nov 01 06:23:02.792723 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:02.792656 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="492a4b92-30d2-438f-b1c4-a7f7c46c56dd" containerName="revoke-keys" Nov 01 06:23:02.796268 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:02.796247 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:23:02.849349 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:02.849310 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c7e551a7-openshift-default-dockercfg-g2jlc\"" Nov 01 06:23:02.903296 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:02.903264 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx"] Nov 01 06:23:02.917969 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:02.917933 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-credential-socket\") pod \"e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx\" (UID: \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\") " pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:23:02.918152 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:02.917976 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-istio-podinfo\") pod \"e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx\" (UID: \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\") " pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:23:02.918152 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:02.918087 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-workload-certs\") pod \"e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx\" (UID: \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\") " pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:23:02.918152 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:02.918116 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-istio-data\") pod \"e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx\" (UID: \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\") " pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:23:02.918282 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:02.918163 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-istio-envoy\") pod \"e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx\" (UID: \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\") " pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:23:02.918317 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:02.918299 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-istiod-ca-cert\") pod \"e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx\" (UID: \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\") " pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:23:02.918351 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:02.918341 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-istio-token\") pod \"e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx\" (UID: \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\") " pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:23:02.918415 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:02.918398 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47csm\" (UniqueName: \"kubernetes.io/projected/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-kube-api-access-47csm\") pod \"e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx\" (UID: \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\") " pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:23:02.918530 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:02.918516 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-workload-socket\") pod \"e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx\" (UID: \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\") " pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:23:03.019544 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:03.019512 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-credential-socket\") pod \"e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx\" (UID: \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\") " pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:23:03.019544 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:03.019550 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-istio-podinfo\") pod \"e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx\" (UID: \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\") " pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:23:03.019806 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:03.019592 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-workload-certs\") pod \"e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx\" (UID: \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\") " pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:23:03.019806 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:03.019611 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-istio-data\") pod \"e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx\" (UID: \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\") " pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:23:03.019806 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:03.019640 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-istio-envoy\") pod \"e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx\" (UID: \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\") " pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:23:03.019806 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:03.019667 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-istiod-ca-cert\") pod \"e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx\" (UID: \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\") " pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:23:03.019806 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:03.019698 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-istio-token\") pod \"e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx\" (UID: \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\") " pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:23:03.019806 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:03.019731 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47csm\" (UniqueName: \"kubernetes.io/projected/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-kube-api-access-47csm\") pod \"e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx\" (UID: \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\") " pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:23:03.019806 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:03.019802 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-workload-socket\") pod \"e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx\" (UID: \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\") " pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:23:03.020188 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:03.019986 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-credential-socket\") pod \"e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx\" (UID: \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\") " pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:23:03.020188 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:03.020027 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-workload-certs\") pod \"e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx\" (UID: \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\") " pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:23:03.020320 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:03.020302 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-istio-data\") pod \"e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx\" (UID: \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\") " pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:23:03.020407 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:03.020385 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-workload-socket\") pod \"e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx\" (UID: \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\") " pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:23:03.020892 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:03.020860 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-istiod-ca-cert\") pod \"e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx\" (UID: \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\") " pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:23:03.022186 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:03.022162 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-istio-envoy\") pod \"e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx\" (UID: \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\") " pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:23:03.022518 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:03.022471 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-istio-podinfo\") pod \"e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx\" (UID: \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\") " pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:23:03.082983 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:03.082872 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-istio-token\") pod \"e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx\" (UID: \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\") " pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:23:03.104657 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:03.104614 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47csm\" (UniqueName: \"kubernetes.io/projected/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-kube-api-access-47csm\") pod \"e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx\" (UID: \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\") " pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:23:03.109881 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:03.109841 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:23:03.552249 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:03.552214 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx"] Nov 01 06:23:03.553322 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:23:03.553290 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0f9327e_b3f9_4af9_b9c9_506a8859a46f.slice/crio-ec8b712a7fec5450a9abbe6838984aa38a69946a9b40b823bf11f5a440abd241 WatchSource:0}: Error finding container ec8b712a7fec5450a9abbe6838984aa38a69946a9b40b823bf11f5a440abd241: Status 404 returned error can't find the container with id ec8b712a7fec5450a9abbe6838984aa38a69946a9b40b823bf11f5a440abd241 Nov 01 06:23:03.555752 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:03.555718 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:23:03.555855 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:03.555782 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:23:03.555855 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:03.555809 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:23:03.634062 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:03.634024 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" event={"ID":"b0f9327e-b3f9-4af9-b9c9-506a8859a46f","Type":"ContainerStarted","Data":"0ebf75d2da3392bd93380e3ce2354794bc3c53dd1aa42ab3274ef8bac9db512d"} Nov 01 06:23:03.634204 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:03.634081 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" event={"ID":"b0f9327e-b3f9-4af9-b9c9-506a8859a46f","Type":"ContainerStarted","Data":"ec8b712a7fec5450a9abbe6838984aa38a69946a9b40b823bf11f5a440abd241"} Nov 01 06:23:03.755558 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:03.755504 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" podStartSLOduration=1.755473906 podStartE2EDuration="1.755473906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:23:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:23:03.754914253 +0000 UTC m=+2262.410031431" watchObservedRunningTime="2026-11-01 06:23:03.755473906 +0000 UTC m=+2262.410591073" Nov 01 06:23:04.110401 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:04.110362 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:23:04.115836 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:04.115811 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:23:04.640318 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:04.640280 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:23:04.641186 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:04.641165 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:23:09.987345 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:23:09.987271 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:23:27.354529 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:27.354457 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5d7100d-5pxzp"] Nov 01 06:23:27.405704 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:27.405669 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5d7100d-5pxzp"] Nov 01 06:23:27.514882 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:27.514852 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="492a4b92-30d2-438f-b1c4-a7f7c46c56dd" path="/var/lib/kubelet/pods/492a4b92-30d2-438f-b1c4-a7f7c46c56dd/volumes" Nov 01 06:23:39.972894 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:23:39.972813 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:23:59.531008 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:59.530974 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7122-eb7f8e"] Nov 01 06:23:59.534807 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:59.534785 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7122-eb7f8e" Nov 01 06:23:59.546455 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:59.546426 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48fh5\" (UniqueName: \"kubernetes.io/projected/5e4f2480-b584-4385-8cb1-0af04b8da9a1-kube-api-access-48fh5\") pod \"test-curl-7122-eb7f8e\" (UID: \"5e4f2480-b584-4385-8cb1-0af04b8da9a1\") " pod="opendatahub/test-curl-7122-eb7f8e" Nov 01 06:23:59.587951 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:59.587926 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7122-eb7f8e"] Nov 01 06:23:59.635159 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:59.634963 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hgn5l\"" Nov 01 06:23:59.648095 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:59.648061 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48fh5\" (UniqueName: \"kubernetes.io/projected/5e4f2480-b584-4385-8cb1-0af04b8da9a1-kube-api-access-48fh5\") pod \"test-curl-7122-eb7f8e\" (UID: \"5e4f2480-b584-4385-8cb1-0af04b8da9a1\") " pod="opendatahub/test-curl-7122-eb7f8e" Nov 01 06:23:59.709250 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:59.709218 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48fh5\" (UniqueName: \"kubernetes.io/projected/5e4f2480-b584-4385-8cb1-0af04b8da9a1-kube-api-access-48fh5\") pod \"test-curl-7122-eb7f8e\" (UID: \"5e4f2480-b584-4385-8cb1-0af04b8da9a1\") " pod="opendatahub/test-curl-7122-eb7f8e" Nov 01 06:23:59.847048 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:23:59.846962 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7122-eb7f8e" Nov 01 06:24:00.223520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:00.222515 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7122-eb7f8e"] Nov 01 06:24:00.257881 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:00.257850 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c69393ab-kl5js"] Nov 01 06:24:00.263718 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:00.263700 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69393ab-kl5js" Nov 01 06:24:00.318464 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:00.318434 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c69393ab-kl5js"] Nov 01 06:24:00.332528 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:00.332490 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:24:00.332697 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:00.332561 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:24:00.354006 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:00.353971 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dc14a58-dd7a-4024-860b-ad7de008bc36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c69393ab-kl5js\" (UID: \"3dc14a58-dd7a-4024-860b-ad7de008bc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69393ab-kl5js" Nov 01 06:24:00.354170 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:00.354038 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dc14a58-dd7a-4024-860b-ad7de008bc36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c69393ab-kl5js\" (UID: \"3dc14a58-dd7a-4024-860b-ad7de008bc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69393ab-kl5js" Nov 01 06:24:00.354218 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:00.354165 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltccn\" (UniqueName: \"kubernetes.io/projected/3dc14a58-dd7a-4024-860b-ad7de008bc36-kube-api-access-ltccn\") pod \"maas-api-revoke-sub-c69393ab-kl5js\" (UID: \"3dc14a58-dd7a-4024-860b-ad7de008bc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69393ab-kl5js" Nov 01 06:24:00.363885 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:00.363842 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:24:00.364012 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:00.363900 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:24:00.455711 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:00.455677 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dc14a58-dd7a-4024-860b-ad7de008bc36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c69393ab-kl5js\" (UID: \"3dc14a58-dd7a-4024-860b-ad7de008bc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69393ab-kl5js" Nov 01 06:24:00.455882 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:00.455756 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltccn\" (UniqueName: \"kubernetes.io/projected/3dc14a58-dd7a-4024-860b-ad7de008bc36-kube-api-access-ltccn\") pod \"maas-api-revoke-sub-c69393ab-kl5js\" (UID: \"3dc14a58-dd7a-4024-860b-ad7de008bc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69393ab-kl5js" Nov 01 06:24:00.455882 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:00.455858 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dc14a58-dd7a-4024-860b-ad7de008bc36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c69393ab-kl5js\" (UID: \"3dc14a58-dd7a-4024-860b-ad7de008bc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69393ab-kl5js" Nov 01 06:24:00.456339 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:00.456311 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dc14a58-dd7a-4024-860b-ad7de008bc36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c69393ab-kl5js\" (UID: \"3dc14a58-dd7a-4024-860b-ad7de008bc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69393ab-kl5js" Nov 01 06:24:00.458203 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:00.458181 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dc14a58-dd7a-4024-860b-ad7de008bc36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c69393ab-kl5js\" (UID: \"3dc14a58-dd7a-4024-860b-ad7de008bc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69393ab-kl5js" Nov 01 06:24:00.517566 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:00.517537 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltccn\" (UniqueName: \"kubernetes.io/projected/3dc14a58-dd7a-4024-860b-ad7de008bc36-kube-api-access-ltccn\") pod \"maas-api-revoke-sub-c69393ab-kl5js\" (UID: \"3dc14a58-dd7a-4024-860b-ad7de008bc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69393ab-kl5js" Nov 01 06:24:00.609401 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:00.609369 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69393ab-kl5js" Nov 01 06:24:00.988126 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:00.988092 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c69393ab-kl5js"] Nov 01 06:24:00.988886 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:24:00.988861 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dc14a58_dd7a_4024_860b_ad7de008bc36.slice/crio-f49a7814b0d72ac551795b5b21a4366cb4065486e1d133e8602c7c50ae614f0d WatchSource:0}: Error finding container f49a7814b0d72ac551795b5b21a4366cb4065486e1d133e8602c7c50ae614f0d: Status 404 returned error can't find the container with id f49a7814b0d72ac551795b5b21a4366cb4065486e1d133e8602c7c50ae614f0d Nov 01 06:24:00.999411 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:00.999378 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7122-eb7f8e" event={"ID":"5e4f2480-b584-4385-8cb1-0af04b8da9a1","Type":"ContainerStarted","Data":"f9e52b6b25f362a3b97a19bf0f0bda8aac6725a4717b749872c0696e6b0414cc"} Nov 01 06:24:00.999543 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:00.999420 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7122-eb7f8e" event={"ID":"5e4f2480-b584-4385-8cb1-0af04b8da9a1","Type":"ContainerStarted","Data":"33847fae12b4903e7f0a2e03957f631bdb4bcc6b08890362a83c39962b82d442"} Nov 01 06:24:01.000819 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:01.000797 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69393ab-kl5js" event={"ID":"3dc14a58-dd7a-4024-860b-ad7de008bc36","Type":"ContainerStarted","Data":"f49a7814b0d72ac551795b5b21a4366cb4065486e1d133e8602c7c50ae614f0d"} Nov 01 06:24:01.116938 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:01.116883 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7122-eb7f8e" podStartSLOduration=2.11686768 podStartE2EDuration="2.11686768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:23:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:24:01.115573054 +0000 UTC m=+2319.770690228" watchObservedRunningTime="2026-11-01 06:24:01.11686768 +0000 UTC m=+2319.771984844" Nov 01 06:24:01.904982 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:24:01.904953 5040 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dc14a58_dd7a_4024_860b_ad7de008bc36.slice/crio-86a27d0a0902b325e907436854e28ea824af1bc1eafc159336eb8e476b5c257f.scope\": RecentStats: unable to find data in memory cache]" Nov 01 06:24:02.009459 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:02.009423 5040 generic.go:356] "Generic (PLEG): container finished" podID="3dc14a58-dd7a-4024-860b-ad7de008bc36" containerID="86a27d0a0902b325e907436854e28ea824af1bc1eafc159336eb8e476b5c257f" exitCode=0 Nov 01 06:24:02.009646 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:02.009507 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69393ab-kl5js" event={"ID":"3dc14a58-dd7a-4024-860b-ad7de008bc36","Type":"ContainerDied","Data":"86a27d0a0902b325e907436854e28ea824af1bc1eafc159336eb8e476b5c257f"} Nov 01 06:24:02.605325 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:02.605290 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7122-eb7f8e"] Nov 01 06:24:02.609675 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:02.609644 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7122-eb7f8e"] Nov 01 06:24:02.872661 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:02.872582 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40f0d38f-dqxbg"] Nov 01 06:24:02.873093 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:02.873079 5040 cpu_manager.go:414] "containerMap: removing container" podUID="5e4f2480-b584-4385-8cb1-0af04b8da9a1" containerName="test-curl-7122-eb7f8e" Nov 01 06:24:02.873141 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:02.873093 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e4f2480-b584-4385-8cb1-0af04b8da9a1" containerName="test-curl-7122-eb7f8e" Nov 01 06:24:02.873187 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:02.873177 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e4f2480-b584-4385-8cb1-0af04b8da9a1" containerName="test-curl-7122-eb7f8e" Nov 01 06:24:02.876262 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:02.876245 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40f0d38f-dqxbg" Nov 01 06:24:02.934438 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:02.934404 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40f0d38f-dqxbg"] Nov 01 06:24:02.987883 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:02.987311 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7q64\" (UniqueName: \"kubernetes.io/projected/6c3629eb-bb73-46e3-b7a9-9447a92cd1c7-kube-api-access-r7q64\") pod \"maas-api-revoke-sub-40f0d38f-dqxbg\" (UID: \"6c3629eb-bb73-46e3-b7a9-9447a92cd1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40f0d38f-dqxbg" Nov 01 06:24:02.988308 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:02.988277 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c3629eb-bb73-46e3-b7a9-9447a92cd1c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40f0d38f-dqxbg\" (UID: \"6c3629eb-bb73-46e3-b7a9-9447a92cd1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40f0d38f-dqxbg" Nov 01 06:24:02.988682 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:02.988660 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c3629eb-bb73-46e3-b7a9-9447a92cd1c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40f0d38f-dqxbg\" (UID: \"6c3629eb-bb73-46e3-b7a9-9447a92cd1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40f0d38f-dqxbg" Nov 01 06:24:03.017930 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:03.017862 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7122-eb7f8e" podUID="5e4f2480-b584-4385-8cb1-0af04b8da9a1" containerName="test-curl-7122-eb7f8e" containerID="cri-o://f9e52b6b25f362a3b97a19bf0f0bda8aac6725a4717b749872c0696e6b0414cc" gracePeriod=2 Nov 01 06:24:03.069785 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:24:03.069739 5040 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7122-eb7f8e\" is forbidden: User \"system:node:ip-10-0-143-195.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-195.us-west-1.compute.internal' and this object" podUID="5e4f2480-b584-4385-8cb1-0af04b8da9a1" pod="opendatahub/test-curl-7122-eb7f8e" Nov 01 06:24:03.090111 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:03.090074 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c3629eb-bb73-46e3-b7a9-9447a92cd1c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40f0d38f-dqxbg\" (UID: \"6c3629eb-bb73-46e3-b7a9-9447a92cd1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40f0d38f-dqxbg" Nov 01 06:24:03.090275 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:03.090138 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7q64\" (UniqueName: \"kubernetes.io/projected/6c3629eb-bb73-46e3-b7a9-9447a92cd1c7-kube-api-access-r7q64\") pod \"maas-api-revoke-sub-40f0d38f-dqxbg\" (UID: \"6c3629eb-bb73-46e3-b7a9-9447a92cd1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40f0d38f-dqxbg" Nov 01 06:24:03.090275 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:03.090263 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c3629eb-bb73-46e3-b7a9-9447a92cd1c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40f0d38f-dqxbg\" (UID: \"6c3629eb-bb73-46e3-b7a9-9447a92cd1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40f0d38f-dqxbg" Nov 01 06:24:03.091008 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:03.090975 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c3629eb-bb73-46e3-b7a9-9447a92cd1c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40f0d38f-dqxbg\" (UID: \"6c3629eb-bb73-46e3-b7a9-9447a92cd1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40f0d38f-dqxbg" Nov 01 06:24:03.092739 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:03.092710 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c3629eb-bb73-46e3-b7a9-9447a92cd1c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40f0d38f-dqxbg\" (UID: \"6c3629eb-bb73-46e3-b7a9-9447a92cd1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40f0d38f-dqxbg" Nov 01 06:24:03.152279 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:03.152246 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7q64\" (UniqueName: \"kubernetes.io/projected/6c3629eb-bb73-46e3-b7a9-9447a92cd1c7-kube-api-access-r7q64\") pod \"maas-api-revoke-sub-40f0d38f-dqxbg\" (UID: \"6c3629eb-bb73-46e3-b7a9-9447a92cd1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40f0d38f-dqxbg" Nov 01 06:24:03.157135 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:03.157109 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69393ab-kl5js" Nov 01 06:24:03.189747 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:03.189723 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40f0d38f-dqxbg" Nov 01 06:24:03.191021 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:03.190999 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dc14a58-dd7a-4024-860b-ad7de008bc36-kube-api-access-ltccn\" (UniqueName: \"kubernetes.io/projected/3dc14a58-dd7a-4024-860b-ad7de008bc36-kube-api-access-ltccn\") pod \"3dc14a58-dd7a-4024-860b-ad7de008bc36\" (UID: \"3dc14a58-dd7a-4024-860b-ad7de008bc36\") " Nov 01 06:24:03.191124 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:03.191035 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dc14a58-dd7a-4024-860b-ad7de008bc36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dc14a58-dd7a-4024-860b-ad7de008bc36-maas-api-openshift-ca\") pod \"3dc14a58-dd7a-4024-860b-ad7de008bc36\" (UID: \"3dc14a58-dd7a-4024-860b-ad7de008bc36\") " Nov 01 06:24:03.191124 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:03.191090 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dc14a58-dd7a-4024-860b-ad7de008bc36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dc14a58-dd7a-4024-860b-ad7de008bc36-maas-api-xks-ca\") pod \"3dc14a58-dd7a-4024-860b-ad7de008bc36\" (UID: \"3dc14a58-dd7a-4024-860b-ad7de008bc36\") " Nov 01 06:24:03.191422 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:03.191398 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dc14a58-dd7a-4024-860b-ad7de008bc36-maas-api-openshift-ca" pod "3dc14a58-dd7a-4024-860b-ad7de008bc36" (UID: "3dc14a58-dd7a-4024-860b-ad7de008bc36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:24:03.193053 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:03.193033 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dc14a58-dd7a-4024-860b-ad7de008bc36-maas-api-xks-ca" pod "3dc14a58-dd7a-4024-860b-ad7de008bc36" (UID: "3dc14a58-dd7a-4024-860b-ad7de008bc36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:24:03.193291 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:03.193271 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dc14a58-dd7a-4024-860b-ad7de008bc36-kube-api-access-ltccn" pod "3dc14a58-dd7a-4024-860b-ad7de008bc36" (UID: "3dc14a58-dd7a-4024-860b-ad7de008bc36"). InnerVolumeSpecName "kube-api-access-ltccn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:24:03.208975 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:24:03.208949 5040 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7122-eb7f8e\" is forbidden: User \"system:node:ip-10-0-143-195.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-195.us-west-1.compute.internal' and this object" podUID="5e4f2480-b584-4385-8cb1-0af04b8da9a1" pod="opendatahub/test-curl-7122-eb7f8e" Nov 01 06:24:03.292780 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:03.292752 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltccn\" (UniqueName: \"kubernetes.io/projected/3dc14a58-dd7a-4024-860b-ad7de008bc36-kube-api-access-ltccn\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:24:03.292780 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:03.292780 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dc14a58-dd7a-4024-860b-ad7de008bc36-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:24:03.292970 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:03.292791 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dc14a58-dd7a-4024-860b-ad7de008bc36-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:24:03.562749 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:03.562721 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40f0d38f-dqxbg"] Nov 01 06:24:03.563955 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:24:03.563928 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c3629eb_bb73_46e3_b7a9_9447a92cd1c7.slice/crio-990e526b6c798e5095d6fa7578b8210a663161f31fd5e942c7e0594e82b63ee0 WatchSource:0}: Error finding container 990e526b6c798e5095d6fa7578b8210a663161f31fd5e942c7e0594e82b63ee0: Status 404 returned error can't find the container with id 990e526b6c798e5095d6fa7578b8210a663161f31fd5e942c7e0594e82b63ee0 Nov 01 06:24:04.024803 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:04.024775 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69393ab-kl5js" Nov 01 06:24:04.025268 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:04.024818 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69393ab-kl5js" event={"ID":"3dc14a58-dd7a-4024-860b-ad7de008bc36","Type":"ContainerDied","Data":"f49a7814b0d72ac551795b5b21a4366cb4065486e1d133e8602c7c50ae614f0d"} Nov 01 06:24:04.025268 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:04.024872 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f49a7814b0d72ac551795b5b21a4366cb4065486e1d133e8602c7c50ae614f0d" Nov 01 06:24:04.026546 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:04.026520 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40f0d38f-dqxbg" event={"ID":"6c3629eb-bb73-46e3-b7a9-9447a92cd1c7","Type":"ContainerStarted","Data":"fa366863a122d0c5cc4a9303cb729cb4f2ab0dcfdda9cdfb0942429339e3e8a9"} Nov 01 06:24:04.026691 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:04.026572 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40f0d38f-dqxbg" event={"ID":"6c3629eb-bb73-46e3-b7a9-9447a92cd1c7","Type":"ContainerStarted","Data":"990e526b6c798e5095d6fa7578b8210a663161f31fd5e942c7e0594e82b63ee0"} Nov 01 06:24:04.146153 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:04.146056 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40f0d38f-dqxbg" podStartSLOduration=2.146043431 podStartE2EDuration="2.146043431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:24:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:24:04.143109887 +0000 UTC m=+2322.798227055" watchObservedRunningTime="2026-11-01 06:24:04.146043431 +0000 UTC m=+2322.801160597" Nov 01 06:24:05.033880 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:05.033847 5040 generic.go:356] "Generic (PLEG): container finished" podID="6c3629eb-bb73-46e3-b7a9-9447a92cd1c7" containerID="fa366863a122d0c5cc4a9303cb729cb4f2ab0dcfdda9cdfb0942429339e3e8a9" exitCode=0 Nov 01 06:24:05.034210 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:05.033893 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40f0d38f-dqxbg" event={"ID":"6c3629eb-bb73-46e3-b7a9-9447a92cd1c7","Type":"ContainerDied","Data":"fa366863a122d0c5cc4a9303cb729cb4f2ab0dcfdda9cdfb0942429339e3e8a9"} Nov 01 06:24:05.231735 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:05.231703 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7122-eb7f8e" Nov 01 06:24:05.314696 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:05.314619 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e4f2480-b584-4385-8cb1-0af04b8da9a1-kube-api-access-48fh5\" (UniqueName: \"kubernetes.io/projected/5e4f2480-b584-4385-8cb1-0af04b8da9a1-kube-api-access-48fh5\") pod \"5e4f2480-b584-4385-8cb1-0af04b8da9a1\" (UID: \"5e4f2480-b584-4385-8cb1-0af04b8da9a1\") " Nov 01 06:24:05.316711 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:05.316680 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e4f2480-b584-4385-8cb1-0af04b8da9a1-kube-api-access-48fh5" pod "5e4f2480-b584-4385-8cb1-0af04b8da9a1" (UID: "5e4f2480-b584-4385-8cb1-0af04b8da9a1"). InnerVolumeSpecName "kube-api-access-48fh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:24:05.416042 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:05.416005 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48fh5\" (UniqueName: \"kubernetes.io/projected/5e4f2480-b584-4385-8cb1-0af04b8da9a1-kube-api-access-48fh5\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:24:05.514293 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:05.514255 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e4f2480-b584-4385-8cb1-0af04b8da9a1" path="/var/lib/kubelet/pods/5e4f2480-b584-4385-8cb1-0af04b8da9a1/volumes" Nov 01 06:24:06.042074 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:06.042036 5040 generic.go:356] "Generic (PLEG): container finished" podID="5e4f2480-b584-4385-8cb1-0af04b8da9a1" containerID="f9e52b6b25f362a3b97a19bf0f0bda8aac6725a4717b749872c0696e6b0414cc" exitCode=137 Nov 01 06:24:06.042513 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:06.042081 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7122-eb7f8e" Nov 01 06:24:06.042513 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:06.042133 5040 scope.go:122] "RemoveContainer" containerID="f9e52b6b25f362a3b97a19bf0f0bda8aac6725a4717b749872c0696e6b0414cc" Nov 01 06:24:06.051526 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:06.051474 5040 scope.go:122] "RemoveContainer" containerID="f9e52b6b25f362a3b97a19bf0f0bda8aac6725a4717b749872c0696e6b0414cc" Nov 01 06:24:06.051868 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:24:06.051848 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9e52b6b25f362a3b97a19bf0f0bda8aac6725a4717b749872c0696e6b0414cc\": container with ID starting with f9e52b6b25f362a3b97a19bf0f0bda8aac6725a4717b749872c0696e6b0414cc not found: ID does not exist" containerID="f9e52b6b25f362a3b97a19bf0f0bda8aac6725a4717b749872c0696e6b0414cc" Nov 01 06:24:06.051953 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:06.051875 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9e52b6b25f362a3b97a19bf0f0bda8aac6725a4717b749872c0696e6b0414cc"} err="failed to get container status \"f9e52b6b25f362a3b97a19bf0f0bda8aac6725a4717b749872c0696e6b0414cc\": rpc error: code = NotFound desc = could not find container \"f9e52b6b25f362a3b97a19bf0f0bda8aac6725a4717b749872c0696e6b0414cc\": container with ID starting with f9e52b6b25f362a3b97a19bf0f0bda8aac6725a4717b749872c0696e6b0414cc not found: ID does not exist" Nov 01 06:24:06.182547 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:06.182516 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40f0d38f-dqxbg" Nov 01 06:24:06.224578 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:06.224544 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c3629eb-bb73-46e3-b7a9-9447a92cd1c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c3629eb-bb73-46e3-b7a9-9447a92cd1c7-maas-api-xks-ca\") pod \"6c3629eb-bb73-46e3-b7a9-9447a92cd1c7\" (UID: \"6c3629eb-bb73-46e3-b7a9-9447a92cd1c7\") " Nov 01 06:24:06.224753 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:06.224601 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c3629eb-bb73-46e3-b7a9-9447a92cd1c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c3629eb-bb73-46e3-b7a9-9447a92cd1c7-maas-api-openshift-ca\") pod \"6c3629eb-bb73-46e3-b7a9-9447a92cd1c7\" (UID: \"6c3629eb-bb73-46e3-b7a9-9447a92cd1c7\") " Nov 01 06:24:06.224753 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:06.224632 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c3629eb-bb73-46e3-b7a9-9447a92cd1c7-kube-api-access-r7q64\" (UniqueName: \"kubernetes.io/projected/6c3629eb-bb73-46e3-b7a9-9447a92cd1c7-kube-api-access-r7q64\") pod \"6c3629eb-bb73-46e3-b7a9-9447a92cd1c7\" (UID: \"6c3629eb-bb73-46e3-b7a9-9447a92cd1c7\") " Nov 01 06:24:06.225021 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:06.224994 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c3629eb-bb73-46e3-b7a9-9447a92cd1c7-maas-api-openshift-ca" pod "6c3629eb-bb73-46e3-b7a9-9447a92cd1c7" (UID: "6c3629eb-bb73-46e3-b7a9-9447a92cd1c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:24:06.227239 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:06.227213 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c3629eb-bb73-46e3-b7a9-9447a92cd1c7-maas-api-xks-ca" pod "6c3629eb-bb73-46e3-b7a9-9447a92cd1c7" (UID: "6c3629eb-bb73-46e3-b7a9-9447a92cd1c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:24:06.227239 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:06.227215 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c3629eb-bb73-46e3-b7a9-9447a92cd1c7-kube-api-access-r7q64" pod "6c3629eb-bb73-46e3-b7a9-9447a92cd1c7" (UID: "6c3629eb-bb73-46e3-b7a9-9447a92cd1c7"). InnerVolumeSpecName "kube-api-access-r7q64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:24:06.326374 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:06.326301 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c3629eb-bb73-46e3-b7a9-9447a92cd1c7-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:24:06.326374 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:06.326327 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c3629eb-bb73-46e3-b7a9-9447a92cd1c7-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:24:06.326374 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:06.326338 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7q64\" (UniqueName: \"kubernetes.io/projected/6c3629eb-bb73-46e3-b7a9-9447a92cd1c7-kube-api-access-r7q64\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:24:07.049611 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:07.049575 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40f0d38f-dqxbg" Nov 01 06:24:07.049611 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:07.049601 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40f0d38f-dqxbg" event={"ID":"6c3629eb-bb73-46e3-b7a9-9447a92cd1c7","Type":"ContainerDied","Data":"990e526b6c798e5095d6fa7578b8210a663161f31fd5e942c7e0594e82b63ee0"} Nov 01 06:24:07.050146 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:07.049657 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="990e526b6c798e5095d6fa7578b8210a663161f31fd5e942c7e0594e82b63ee0" Nov 01 06:24:09.971776 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:24:09.971732 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:24:12.475973 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:12.475928 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-489da514-7c9d7"] Nov 01 06:24:12.476596 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:12.476569 5040 cpu_manager.go:414] "containerMap: removing container" podUID="6c3629eb-bb73-46e3-b7a9-9447a92cd1c7" containerName="revoke-keys" Nov 01 06:24:12.476596 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:12.476590 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c3629eb-bb73-46e3-b7a9-9447a92cd1c7" containerName="revoke-keys" Nov 01 06:24:12.476793 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:12.476624 5040 cpu_manager.go:414] "containerMap: removing container" podUID="3dc14a58-dd7a-4024-860b-ad7de008bc36" containerName="revoke-keys" Nov 01 06:24:12.476793 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:12.476633 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dc14a58-dd7a-4024-860b-ad7de008bc36" containerName="revoke-keys" Nov 01 06:24:12.476793 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:12.476751 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c3629eb-bb73-46e3-b7a9-9447a92cd1c7" containerName="revoke-keys" Nov 01 06:24:12.476793 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:12.476769 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dc14a58-dd7a-4024-860b-ad7de008bc36" containerName="revoke-keys" Nov 01 06:24:12.480547 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:12.480525 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-489da514-7c9d7" Nov 01 06:24:12.536835 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:12.536803 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-489da514-7c9d7"] Nov 01 06:24:12.582882 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:12.582853 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:24:12.583044 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:12.582883 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:24:12.583044 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:12.582893 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:24:12.583878 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:12.583854 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:24:12.595995 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:12.595974 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb003776-c6cd-4f28-966e-9c3b91dd19cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-489da514-7c9d7\" (UID: \"bb003776-c6cd-4f28-966e-9c3b91dd19cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-489da514-7c9d7" Nov 01 06:24:12.596080 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:12.596021 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb003776-c6cd-4f28-966e-9c3b91dd19cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-489da514-7c9d7\" (UID: \"bb003776-c6cd-4f28-966e-9c3b91dd19cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-489da514-7c9d7" Nov 01 06:24:12.596168 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:12.596156 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kjvb\" (UniqueName: \"kubernetes.io/projected/bb003776-c6cd-4f28-966e-9c3b91dd19cd-kube-api-access-9kjvb\") pod \"maas-api-revoke-sub-489da514-7c9d7\" (UID: \"bb003776-c6cd-4f28-966e-9c3b91dd19cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-489da514-7c9d7" Nov 01 06:24:12.697422 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:12.697391 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb003776-c6cd-4f28-966e-9c3b91dd19cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-489da514-7c9d7\" (UID: \"bb003776-c6cd-4f28-966e-9c3b91dd19cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-489da514-7c9d7" Nov 01 06:24:12.697593 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:12.697448 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb003776-c6cd-4f28-966e-9c3b91dd19cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-489da514-7c9d7\" (UID: \"bb003776-c6cd-4f28-966e-9c3b91dd19cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-489da514-7c9d7" Nov 01 06:24:12.697593 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:12.697524 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kjvb\" (UniqueName: \"kubernetes.io/projected/bb003776-c6cd-4f28-966e-9c3b91dd19cd-kube-api-access-9kjvb\") pod \"maas-api-revoke-sub-489da514-7c9d7\" (UID: \"bb003776-c6cd-4f28-966e-9c3b91dd19cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-489da514-7c9d7" Nov 01 06:24:12.698111 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:12.698087 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb003776-c6cd-4f28-966e-9c3b91dd19cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-489da514-7c9d7\" (UID: \"bb003776-c6cd-4f28-966e-9c3b91dd19cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-489da514-7c9d7" Nov 01 06:24:12.699804 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:12.699784 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb003776-c6cd-4f28-966e-9c3b91dd19cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-489da514-7c9d7\" (UID: \"bb003776-c6cd-4f28-966e-9c3b91dd19cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-489da514-7c9d7" Nov 01 06:24:12.759182 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:12.759147 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kjvb\" (UniqueName: \"kubernetes.io/projected/bb003776-c6cd-4f28-966e-9c3b91dd19cd-kube-api-access-9kjvb\") pod \"maas-api-revoke-sub-489da514-7c9d7\" (UID: \"bb003776-c6cd-4f28-966e-9c3b91dd19cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-489da514-7c9d7" Nov 01 06:24:12.795721 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:12.795695 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-489da514-7c9d7" Nov 01 06:24:13.171051 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:13.171025 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-489da514-7c9d7"] Nov 01 06:24:13.172216 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:24:13.172189 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb003776_c6cd_4f28_966e_9c3b91dd19cd.slice/crio-754d0f600831daf187b499e16d8bd9eb497d3c64ce497e2a6ba196ab2bcd7228 WatchSource:0}: Error finding container 754d0f600831daf187b499e16d8bd9eb497d3c64ce497e2a6ba196ab2bcd7228: Status 404 returned error can't find the container with id 754d0f600831daf187b499e16d8bd9eb497d3c64ce497e2a6ba196ab2bcd7228 Nov 01 06:24:13.602293 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:13.602259 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdfb2df7-qvfqw"] Nov 01 06:24:13.606163 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:13.606143 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdfb2df7-qvfqw" Nov 01 06:24:13.704841 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:13.704804 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdfb2df7-qvfqw"] Nov 01 06:24:13.707814 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:13.707782 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41406be2-24cd-449c-90dc-a56a1d07cb90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdfb2df7-qvfqw\" (UID: \"41406be2-24cd-449c-90dc-a56a1d07cb90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdfb2df7-qvfqw" Nov 01 06:24:13.707965 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:13.707853 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd7mx\" (UniqueName: \"kubernetes.io/projected/41406be2-24cd-449c-90dc-a56a1d07cb90-kube-api-access-nd7mx\") pod \"maas-api-revoke-sub-fdfb2df7-qvfqw\" (UID: \"41406be2-24cd-449c-90dc-a56a1d07cb90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdfb2df7-qvfqw" Nov 01 06:24:13.708037 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:13.708002 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41406be2-24cd-449c-90dc-a56a1d07cb90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdfb2df7-qvfqw\" (UID: \"41406be2-24cd-449c-90dc-a56a1d07cb90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdfb2df7-qvfqw" Nov 01 06:24:13.809083 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:13.809002 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41406be2-24cd-449c-90dc-a56a1d07cb90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdfb2df7-qvfqw\" (UID: \"41406be2-24cd-449c-90dc-a56a1d07cb90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdfb2df7-qvfqw" Nov 01 06:24:13.809234 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:13.809107 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41406be2-24cd-449c-90dc-a56a1d07cb90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdfb2df7-qvfqw\" (UID: \"41406be2-24cd-449c-90dc-a56a1d07cb90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdfb2df7-qvfqw" Nov 01 06:24:13.809234 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:13.809140 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd7mx\" (UniqueName: \"kubernetes.io/projected/41406be2-24cd-449c-90dc-a56a1d07cb90-kube-api-access-nd7mx\") pod \"maas-api-revoke-sub-fdfb2df7-qvfqw\" (UID: \"41406be2-24cd-449c-90dc-a56a1d07cb90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdfb2df7-qvfqw" Nov 01 06:24:13.809712 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:13.809685 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41406be2-24cd-449c-90dc-a56a1d07cb90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdfb2df7-qvfqw\" (UID: \"41406be2-24cd-449c-90dc-a56a1d07cb90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdfb2df7-qvfqw" Nov 01 06:24:13.811368 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:13.811349 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41406be2-24cd-449c-90dc-a56a1d07cb90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdfb2df7-qvfqw\" (UID: \"41406be2-24cd-449c-90dc-a56a1d07cb90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdfb2df7-qvfqw" Nov 01 06:24:13.872590 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:13.872565 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd7mx\" (UniqueName: \"kubernetes.io/projected/41406be2-24cd-449c-90dc-a56a1d07cb90-kube-api-access-nd7mx\") pod \"maas-api-revoke-sub-fdfb2df7-qvfqw\" (UID: \"41406be2-24cd-449c-90dc-a56a1d07cb90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdfb2df7-qvfqw" Nov 01 06:24:13.919654 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:13.919618 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdfb2df7-qvfqw" Nov 01 06:24:14.095672 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:14.095638 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-489da514-7c9d7" event={"ID":"bb003776-c6cd-4f28-966e-9c3b91dd19cd","Type":"ContainerStarted","Data":"2f05ad9a69e7c18c9012fde3fe410d58abdd5b786ed42fd1c92e3a194e2a7957"} Nov 01 06:24:14.095882 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:14.095693 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-489da514-7c9d7" event={"ID":"bb003776-c6cd-4f28-966e-9c3b91dd19cd","Type":"ContainerStarted","Data":"754d0f600831daf187b499e16d8bd9eb497d3c64ce497e2a6ba196ab2bcd7228"} Nov 01 06:24:14.216553 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:14.216470 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-489da514-7c9d7" podStartSLOduration=2.216457905 podStartE2EDuration="2.216457905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:24:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:24:14.213198164 +0000 UTC m=+2332.868315331" watchObservedRunningTime="2026-11-01 06:24:14.216457905 +0000 UTC m=+2332.871575132" Nov 01 06:24:14.327847 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:14.327818 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdfb2df7-qvfqw"] Nov 01 06:24:14.328570 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:24:14.328543 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41406be2_24cd_449c_90dc_a56a1d07cb90.slice/crio-3e8f8a98a7b60f45e3b08e728502f0cdacc3e525a7600a7ad71fd784092f907f WatchSource:0}: Error finding container 3e8f8a98a7b60f45e3b08e728502f0cdacc3e525a7600a7ad71fd784092f907f: Status 404 returned error can't find the container with id 3e8f8a98a7b60f45e3b08e728502f0cdacc3e525a7600a7ad71fd784092f907f Nov 01 06:24:15.111995 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:15.111961 5040 generic.go:356] "Generic (PLEG): container finished" podID="bb003776-c6cd-4f28-966e-9c3b91dd19cd" containerID="2f05ad9a69e7c18c9012fde3fe410d58abdd5b786ed42fd1c92e3a194e2a7957" exitCode=0 Nov 01 06:24:15.112446 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:15.112043 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-489da514-7c9d7" event={"ID":"bb003776-c6cd-4f28-966e-9c3b91dd19cd","Type":"ContainerDied","Data":"2f05ad9a69e7c18c9012fde3fe410d58abdd5b786ed42fd1c92e3a194e2a7957"} Nov 01 06:24:15.113901 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:15.113874 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdfb2df7-qvfqw" event={"ID":"41406be2-24cd-449c-90dc-a56a1d07cb90","Type":"ContainerStarted","Data":"07e441a980a384d081582a0589e20364ca5014d55becb97854664f3c3f3cf182"} Nov 01 06:24:15.114018 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:15.113922 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdfb2df7-qvfqw" event={"ID":"41406be2-24cd-449c-90dc-a56a1d07cb90","Type":"ContainerStarted","Data":"3e8f8a98a7b60f45e3b08e728502f0cdacc3e525a7600a7ad71fd784092f907f"} Nov 01 06:24:15.347798 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:15.347731 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdfb2df7-qvfqw" podStartSLOduration=2.347717644 podStartE2EDuration="2.347717644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:24:15.347047165 +0000 UTC m=+2334.002164333" watchObservedRunningTime="2026-11-01 06:24:15.347717644 +0000 UTC m=+2334.002834810" Nov 01 06:24:16.122014 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:16.121978 5040 generic.go:356] "Generic (PLEG): container finished" podID="41406be2-24cd-449c-90dc-a56a1d07cb90" containerID="07e441a980a384d081582a0589e20364ca5014d55becb97854664f3c3f3cf182" exitCode=0 Nov 01 06:24:16.122419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:16.122059 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdfb2df7-qvfqw" event={"ID":"41406be2-24cd-449c-90dc-a56a1d07cb90","Type":"ContainerDied","Data":"07e441a980a384d081582a0589e20364ca5014d55becb97854664f3c3f3cf182"} Nov 01 06:24:16.285750 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:16.285720 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-489da514-7c9d7" Nov 01 06:24:16.334408 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:16.334378 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb003776-c6cd-4f28-966e-9c3b91dd19cd-kube-api-access-9kjvb\" (UniqueName: \"kubernetes.io/projected/bb003776-c6cd-4f28-966e-9c3b91dd19cd-kube-api-access-9kjvb\") pod \"bb003776-c6cd-4f28-966e-9c3b91dd19cd\" (UID: \"bb003776-c6cd-4f28-966e-9c3b91dd19cd\") " Nov 01 06:24:16.334625 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:16.334443 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb003776-c6cd-4f28-966e-9c3b91dd19cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb003776-c6cd-4f28-966e-9c3b91dd19cd-maas-api-openshift-ca\") pod \"bb003776-c6cd-4f28-966e-9c3b91dd19cd\" (UID: \"bb003776-c6cd-4f28-966e-9c3b91dd19cd\") " Nov 01 06:24:16.334625 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:16.334512 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb003776-c6cd-4f28-966e-9c3b91dd19cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb003776-c6cd-4f28-966e-9c3b91dd19cd-maas-api-xks-ca\") pod \"bb003776-c6cd-4f28-966e-9c3b91dd19cd\" (UID: \"bb003776-c6cd-4f28-966e-9c3b91dd19cd\") " Nov 01 06:24:16.334869 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:16.334833 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb003776-c6cd-4f28-966e-9c3b91dd19cd-maas-api-openshift-ca" pod "bb003776-c6cd-4f28-966e-9c3b91dd19cd" (UID: "bb003776-c6cd-4f28-966e-9c3b91dd19cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:24:16.335043 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:16.335019 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb003776-c6cd-4f28-966e-9c3b91dd19cd-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:24:16.336591 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:16.336563 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb003776-c6cd-4f28-966e-9c3b91dd19cd-kube-api-access-9kjvb" pod "bb003776-c6cd-4f28-966e-9c3b91dd19cd" (UID: "bb003776-c6cd-4f28-966e-9c3b91dd19cd"). InnerVolumeSpecName "kube-api-access-9kjvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:24:16.336675 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:16.336602 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb003776-c6cd-4f28-966e-9c3b91dd19cd-maas-api-xks-ca" pod "bb003776-c6cd-4f28-966e-9c3b91dd19cd" (UID: "bb003776-c6cd-4f28-966e-9c3b91dd19cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:24:16.436514 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:16.436413 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kjvb\" (UniqueName: \"kubernetes.io/projected/bb003776-c6cd-4f28-966e-9c3b91dd19cd-kube-api-access-9kjvb\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:24:16.436514 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:16.436447 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb003776-c6cd-4f28-966e-9c3b91dd19cd-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:24:17.129708 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:17.129673 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-489da514-7c9d7" Nov 01 06:24:17.129708 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:17.129705 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-489da514-7c9d7" event={"ID":"bb003776-c6cd-4f28-966e-9c3b91dd19cd","Type":"ContainerDied","Data":"754d0f600831daf187b499e16d8bd9eb497d3c64ce497e2a6ba196ab2bcd7228"} Nov 01 06:24:17.130145 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:17.129748 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="754d0f600831daf187b499e16d8bd9eb497d3c64ce497e2a6ba196ab2bcd7228" Nov 01 06:24:17.304507 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:17.304453 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdfb2df7-qvfqw" Nov 01 06:24:17.447775 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:17.447694 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41406be2-24cd-449c-90dc-a56a1d07cb90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41406be2-24cd-449c-90dc-a56a1d07cb90-maas-api-xks-ca\") pod \"41406be2-24cd-449c-90dc-a56a1d07cb90\" (UID: \"41406be2-24cd-449c-90dc-a56a1d07cb90\") " Nov 01 06:24:17.447775 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:17.447756 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41406be2-24cd-449c-90dc-a56a1d07cb90-kube-api-access-nd7mx\" (UniqueName: \"kubernetes.io/projected/41406be2-24cd-449c-90dc-a56a1d07cb90-kube-api-access-nd7mx\") pod \"41406be2-24cd-449c-90dc-a56a1d07cb90\" (UID: \"41406be2-24cd-449c-90dc-a56a1d07cb90\") " Nov 01 06:24:17.447997 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:17.447785 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41406be2-24cd-449c-90dc-a56a1d07cb90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41406be2-24cd-449c-90dc-a56a1d07cb90-maas-api-openshift-ca\") pod \"41406be2-24cd-449c-90dc-a56a1d07cb90\" (UID: \"41406be2-24cd-449c-90dc-a56a1d07cb90\") " Nov 01 06:24:17.448238 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:17.448205 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41406be2-24cd-449c-90dc-a56a1d07cb90-maas-api-openshift-ca" pod "41406be2-24cd-449c-90dc-a56a1d07cb90" (UID: "41406be2-24cd-449c-90dc-a56a1d07cb90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:24:17.448367 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:17.448346 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41406be2-24cd-449c-90dc-a56a1d07cb90-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:24:17.450107 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:17.450081 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41406be2-24cd-449c-90dc-a56a1d07cb90-maas-api-xks-ca" pod "41406be2-24cd-449c-90dc-a56a1d07cb90" (UID: "41406be2-24cd-449c-90dc-a56a1d07cb90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:24:17.450309 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:17.450288 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41406be2-24cd-449c-90dc-a56a1d07cb90-kube-api-access-nd7mx" pod "41406be2-24cd-449c-90dc-a56a1d07cb90" (UID: "41406be2-24cd-449c-90dc-a56a1d07cb90"). InnerVolumeSpecName "kube-api-access-nd7mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:24:17.548968 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:17.548928 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41406be2-24cd-449c-90dc-a56a1d07cb90-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:24:17.548968 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:17.548958 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd7mx\" (UniqueName: \"kubernetes.io/projected/41406be2-24cd-449c-90dc-a56a1d07cb90-kube-api-access-nd7mx\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:24:18.137423 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:18.137387 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdfb2df7-qvfqw" Nov 01 06:24:18.137423 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:18.137405 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdfb2df7-qvfqw" event={"ID":"41406be2-24cd-449c-90dc-a56a1d07cb90","Type":"ContainerDied","Data":"3e8f8a98a7b60f45e3b08e728502f0cdacc3e525a7600a7ad71fd784092f907f"} Nov 01 06:24:18.137863 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:18.137453 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e8f8a98a7b60f45e3b08e728502f0cdacc3e525a7600a7ad71fd784092f907f" Nov 01 06:24:27.686217 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:27.686180 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52"] Nov 01 06:24:34.186194 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:34.186156 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5507e36a-c2jzw"] Nov 01 06:24:34.186680 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:34.186665 5040 cpu_manager.go:414] "containerMap: removing container" podUID="41406be2-24cd-449c-90dc-a56a1d07cb90" containerName="revoke-keys" Nov 01 06:24:34.186725 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:34.186680 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41406be2-24cd-449c-90dc-a56a1d07cb90" containerName="revoke-keys" Nov 01 06:24:34.186725 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:34.186695 5040 cpu_manager.go:414] "containerMap: removing container" podUID="bb003776-c6cd-4f28-966e-9c3b91dd19cd" containerName="revoke-keys" Nov 01 06:24:34.186725 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:34.186701 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb003776-c6cd-4f28-966e-9c3b91dd19cd" containerName="revoke-keys" Nov 01 06:24:34.186825 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:34.186815 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="41406be2-24cd-449c-90dc-a56a1d07cb90" containerName="revoke-keys" Nov 01 06:24:34.186825 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:34.186824 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb003776-c6cd-4f28-966e-9c3b91dd19cd" containerName="revoke-keys" Nov 01 06:24:34.191524 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:34.191505 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5507e36a-c2jzw" Nov 01 06:24:34.247814 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:34.247785 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5507e36a-c2jzw"] Nov 01 06:24:34.256361 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:34.256322 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lj6r\" (UniqueName: \"kubernetes.io/projected/8b1e622d-5b73-4c97-97eb-7bd89df19ca2-kube-api-access-9lj6r\") pod \"maas-api-revoke-sub-5507e36a-c2jzw\" (UID: \"8b1e622d-5b73-4c97-97eb-7bd89df19ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5507e36a-c2jzw" Nov 01 06:24:34.256554 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:34.256520 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b1e622d-5b73-4c97-97eb-7bd89df19ca2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5507e36a-c2jzw\" (UID: \"8b1e622d-5b73-4c97-97eb-7bd89df19ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5507e36a-c2jzw" Nov 01 06:24:34.256627 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:34.256588 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b1e622d-5b73-4c97-97eb-7bd89df19ca2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5507e36a-c2jzw\" (UID: \"8b1e622d-5b73-4c97-97eb-7bd89df19ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5507e36a-c2jzw" Nov 01 06:24:34.294148 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:34.294108 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:24:34.294325 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:34.294195 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:24:34.329305 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:34.329278 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:24:34.329305 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:34.329286 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:24:34.357597 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:34.357565 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b1e622d-5b73-4c97-97eb-7bd89df19ca2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5507e36a-c2jzw\" (UID: \"8b1e622d-5b73-4c97-97eb-7bd89df19ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5507e36a-c2jzw" Nov 01 06:24:34.357765 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:34.357620 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b1e622d-5b73-4c97-97eb-7bd89df19ca2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5507e36a-c2jzw\" (UID: \"8b1e622d-5b73-4c97-97eb-7bd89df19ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5507e36a-c2jzw" Nov 01 06:24:34.357765 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:34.357698 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lj6r\" (UniqueName: \"kubernetes.io/projected/8b1e622d-5b73-4c97-97eb-7bd89df19ca2-kube-api-access-9lj6r\") pod \"maas-api-revoke-sub-5507e36a-c2jzw\" (UID: \"8b1e622d-5b73-4c97-97eb-7bd89df19ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5507e36a-c2jzw" Nov 01 06:24:34.358390 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:34.358367 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b1e622d-5b73-4c97-97eb-7bd89df19ca2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5507e36a-c2jzw\" (UID: \"8b1e622d-5b73-4c97-97eb-7bd89df19ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5507e36a-c2jzw" Nov 01 06:24:34.360101 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:34.360073 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b1e622d-5b73-4c97-97eb-7bd89df19ca2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5507e36a-c2jzw\" (UID: \"8b1e622d-5b73-4c97-97eb-7bd89df19ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5507e36a-c2jzw" Nov 01 06:24:34.419983 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:34.419946 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lj6r\" (UniqueName: \"kubernetes.io/projected/8b1e622d-5b73-4c97-97eb-7bd89df19ca2-kube-api-access-9lj6r\") pod \"maas-api-revoke-sub-5507e36a-c2jzw\" (UID: \"8b1e622d-5b73-4c97-97eb-7bd89df19ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5507e36a-c2jzw" Nov 01 06:24:34.504951 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:34.504919 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5507e36a-c2jzw" Nov 01 06:24:34.716111 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:34.716073 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4"] Nov 01 06:24:34.892522 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:34.886655 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5507e36a-c2jzw"] Nov 01 06:24:34.892522 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:24:34.888076 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b1e622d_5b73_4c97_97eb_7bd89df19ca2.slice/crio-c96cbf1ae4ea14be33c7eb8a4d61ad2f679e5292e4ca3c37e0a7d103890feff4 WatchSource:0}: Error finding container c96cbf1ae4ea14be33c7eb8a4d61ad2f679e5292e4ca3c37e0a7d103890feff4: Status 404 returned error can't find the container with id c96cbf1ae4ea14be33c7eb8a4d61ad2f679e5292e4ca3c37e0a7d103890feff4 Nov 01 06:24:35.140123 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:35.140068 5040 patch_prober.go:29] interesting pod/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": read tcp 10.132.0.2:39524->10.132.0.155:15021: read: connection reset by peer" start-of-body= Nov 01 06:24:35.140312 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:35.140143 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" podUID="b0f9327e-b3f9-4af9-b9c9-506a8859a46f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": read tcp 10.132.0.2:39524->10.132.0.155:15021: read: connection reset by peer" Nov 01 06:24:35.268692 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:35.268649 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5507e36a-c2jzw" event={"ID":"8b1e622d-5b73-4c97-97eb-7bd89df19ca2","Type":"ContainerStarted","Data":"4c5e3a82f22ee8291cbaf4ca5d2b212439236a9c910784036c2a42c812250358"} Nov 01 06:24:35.269156 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:35.268710 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5507e36a-c2jzw" event={"ID":"8b1e622d-5b73-4c97-97eb-7bd89df19ca2","Type":"ContainerStarted","Data":"c96cbf1ae4ea14be33c7eb8a4d61ad2f679e5292e4ca3c37e0a7d103890feff4"} Nov 01 06:24:35.273537 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:35.273447 5040 generic.go:356] "Generic (PLEG): container finished" podID="b0f9327e-b3f9-4af9-b9c9-506a8859a46f" containerID="0ebf75d2da3392bd93380e3ce2354794bc3c53dd1aa42ab3274ef8bac9db512d" exitCode=0 Nov 01 06:24:35.273714 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:35.273562 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" event={"ID":"b0f9327e-b3f9-4af9-b9c9-506a8859a46f","Type":"ContainerDied","Data":"0ebf75d2da3392bd93380e3ce2354794bc3c53dd1aa42ab3274ef8bac9db512d"} Nov 01 06:24:35.274033 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:35.274006 5040 scope.go:122] "RemoveContainer" containerID="0ebf75d2da3392bd93380e3ce2354794bc3c53dd1aa42ab3274ef8bac9db512d" Nov 01 06:24:35.275771 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:35.275738 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:24:35.275894 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:35.275829 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:24:35.275894 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:35.275879 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:24:35.393234 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:35.393121 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5507e36a-c2jzw" podStartSLOduration=1.393106859 podStartE2EDuration="1.393106859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:24:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:24:35.390178636 +0000 UTC m=+2354.045295807" watchObservedRunningTime="2026-11-01 06:24:35.393106859 +0000 UTC m=+2354.048224060" Nov 01 06:24:36.286503 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:36.286444 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" event={"ID":"b0f9327e-b3f9-4af9-b9c9-506a8859a46f","Type":"ContainerStarted","Data":"46ed39363028c516cda78e8d043de47223dcb31f369ab51217d74c38a439aca8"} Nov 01 06:24:36.288336 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:36.288310 5040 generic.go:356] "Generic (PLEG): container finished" podID="8b1e622d-5b73-4c97-97eb-7bd89df19ca2" containerID="4c5e3a82f22ee8291cbaf4ca5d2b212439236a9c910784036c2a42c812250358" exitCode=0 Nov 01 06:24:36.288491 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:36.288380 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5507e36a-c2jzw" event={"ID":"8b1e622d-5b73-4c97-97eb-7bd89df19ca2","Type":"ContainerDied","Data":"4c5e3a82f22ee8291cbaf4ca5d2b212439236a9c910784036c2a42c812250358"} Nov 01 06:24:37.112505 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:37.111590 5040 patch_prober.go:29] interesting pod/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 01 06:24:37.112505 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:37.111659 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" podUID="b0f9327e-b3f9-4af9-b9c9-506a8859a46f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 01 06:24:37.112505 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:37.111966 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:24:37.529020 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:37.528991 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5507e36a-c2jzw" Nov 01 06:24:37.621973 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:37.621931 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b1e622d-5b73-4c97-97eb-7bd89df19ca2-kube-api-access-9lj6r\" (UniqueName: \"kubernetes.io/projected/8b1e622d-5b73-4c97-97eb-7bd89df19ca2-kube-api-access-9lj6r\") pod \"8b1e622d-5b73-4c97-97eb-7bd89df19ca2\" (UID: \"8b1e622d-5b73-4c97-97eb-7bd89df19ca2\") " Nov 01 06:24:37.622212 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:37.621985 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b1e622d-5b73-4c97-97eb-7bd89df19ca2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b1e622d-5b73-4c97-97eb-7bd89df19ca2-maas-api-xks-ca\") pod \"8b1e622d-5b73-4c97-97eb-7bd89df19ca2\" (UID: \"8b1e622d-5b73-4c97-97eb-7bd89df19ca2\") " Nov 01 06:24:37.622212 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:37.622070 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b1e622d-5b73-4c97-97eb-7bd89df19ca2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b1e622d-5b73-4c97-97eb-7bd89df19ca2-maas-api-openshift-ca\") pod \"8b1e622d-5b73-4c97-97eb-7bd89df19ca2\" (UID: \"8b1e622d-5b73-4c97-97eb-7bd89df19ca2\") " Nov 01 06:24:37.622758 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:37.622723 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b1e622d-5b73-4c97-97eb-7bd89df19ca2-maas-api-openshift-ca" pod "8b1e622d-5b73-4c97-97eb-7bd89df19ca2" (UID: "8b1e622d-5b73-4c97-97eb-7bd89df19ca2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:24:37.624559 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:37.624424 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b1e622d-5b73-4c97-97eb-7bd89df19ca2-maas-api-xks-ca" pod "8b1e622d-5b73-4c97-97eb-7bd89df19ca2" (UID: "8b1e622d-5b73-4c97-97eb-7bd89df19ca2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:24:37.625081 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:37.625055 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b1e622d-5b73-4c97-97eb-7bd89df19ca2-kube-api-access-9lj6r" pod "8b1e622d-5b73-4c97-97eb-7bd89df19ca2" (UID: "8b1e622d-5b73-4c97-97eb-7bd89df19ca2"). InnerVolumeSpecName "kube-api-access-9lj6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:24:37.723514 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:37.723440 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b1e622d-5b73-4c97-97eb-7bd89df19ca2-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:24:37.723514 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:37.723473 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lj6r\" (UniqueName: \"kubernetes.io/projected/8b1e622d-5b73-4c97-97eb-7bd89df19ca2-kube-api-access-9lj6r\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:24:37.723514 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:37.723510 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b1e622d-5b73-4c97-97eb-7bd89df19ca2-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:24:38.112991 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:38.112949 5040 patch_prober.go:29] interesting pod/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 01 06:24:38.113317 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:38.113278 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" podUID="b0f9327e-b3f9-4af9-b9c9-506a8859a46f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 01 06:24:38.324845 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:38.324804 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5507e36a-c2jzw" event={"ID":"8b1e622d-5b73-4c97-97eb-7bd89df19ca2","Type":"ContainerDied","Data":"c96cbf1ae4ea14be33c7eb8a4d61ad2f679e5292e4ca3c37e0a7d103890feff4"} Nov 01 06:24:38.325049 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:38.324876 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c96cbf1ae4ea14be33c7eb8a4d61ad2f679e5292e4ca3c37e0a7d103890feff4" Nov 01 06:24:38.325049 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:38.324953 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5507e36a-c2jzw" Nov 01 06:24:39.118658 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:39.118614 5040 patch_prober.go:29] interesting pod/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 01 06:24:39.119064 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:39.118684 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" podUID="b0f9327e-b3f9-4af9-b9c9-506a8859a46f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 01 06:24:39.973583 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:24:39.973544 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:24:40.113396 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:40.112886 5040 patch_prober.go:29] interesting pod/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 01 06:24:40.113396 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:40.112966 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" podUID="b0f9327e-b3f9-4af9-b9c9-506a8859a46f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 01 06:24:41.112948 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:41.112902 5040 patch_prober.go:29] interesting pod/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 01 06:24:41.113595 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:41.113559 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" podUID="b0f9327e-b3f9-4af9-b9c9-506a8859a46f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 01 06:24:41.629826 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:41.629793 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt"] Nov 01 06:24:41.630288 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:41.630275 5040 cpu_manager.go:414] "containerMap: removing container" podUID="8b1e622d-5b73-4c97-97eb-7bd89df19ca2" containerName="revoke-keys" Nov 01 06:24:41.630333 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:41.630289 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b1e622d-5b73-4c97-97eb-7bd89df19ca2" containerName="revoke-keys" Nov 01 06:24:41.630390 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:41.630381 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b1e622d-5b73-4c97-97eb-7bd89df19ca2" containerName="revoke-keys" Nov 01 06:24:41.634054 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:41.634035 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:24:41.670513 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:41.667505 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/068716fa-4615-4dbf-9c5f-41687865d5b1-credential-socket\") pod \"e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt\" (UID: \"068716fa-4615-4dbf-9c5f-41687865d5b1\") " pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:24:41.670513 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:41.667594 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/068716fa-4615-4dbf-9c5f-41687865d5b1-istiod-ca-cert\") pod \"e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt\" (UID: \"068716fa-4615-4dbf-9c5f-41687865d5b1\") " pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:24:41.670513 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:41.667664 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/068716fa-4615-4dbf-9c5f-41687865d5b1-workload-socket\") pod \"e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt\" (UID: \"068716fa-4615-4dbf-9c5f-41687865d5b1\") " pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:24:41.670513 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:41.667696 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/068716fa-4615-4dbf-9c5f-41687865d5b1-istio-envoy\") pod \"e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt\" (UID: \"068716fa-4615-4dbf-9c5f-41687865d5b1\") " pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:24:41.670513 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:41.667747 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/068716fa-4615-4dbf-9c5f-41687865d5b1-istio-data\") pod \"e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt\" (UID: \"068716fa-4615-4dbf-9c5f-41687865d5b1\") " pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:24:41.670513 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:41.667815 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/068716fa-4615-4dbf-9c5f-41687865d5b1-istio-podinfo\") pod \"e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt\" (UID: \"068716fa-4615-4dbf-9c5f-41687865d5b1\") " pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:24:41.670513 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:41.667849 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/068716fa-4615-4dbf-9c5f-41687865d5b1-istio-token\") pod \"e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt\" (UID: \"068716fa-4615-4dbf-9c5f-41687865d5b1\") " pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:24:41.670513 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:41.667908 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9btp\" (UniqueName: \"kubernetes.io/projected/068716fa-4615-4dbf-9c5f-41687865d5b1-kube-api-access-l9btp\") pod \"e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt\" (UID: \"068716fa-4615-4dbf-9c5f-41687865d5b1\") " pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:24:41.670513 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:41.667959 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/068716fa-4615-4dbf-9c5f-41687865d5b1-workload-certs\") pod \"e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt\" (UID: \"068716fa-4615-4dbf-9c5f-41687865d5b1\") " pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:24:41.692963 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:41.692668 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f13905-openshift-default-dockercfg-8s2fk\"" Nov 01 06:24:41.749170 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:41.749113 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt"] Nov 01 06:24:41.768800 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:41.768762 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/068716fa-4615-4dbf-9c5f-41687865d5b1-istiod-ca-cert\") pod \"e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt\" (UID: \"068716fa-4615-4dbf-9c5f-41687865d5b1\") " pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:24:41.768989 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:41.768841 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/068716fa-4615-4dbf-9c5f-41687865d5b1-workload-socket\") pod \"e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt\" (UID: \"068716fa-4615-4dbf-9c5f-41687865d5b1\") " pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:24:41.768989 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:41.768869 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/068716fa-4615-4dbf-9c5f-41687865d5b1-istio-envoy\") pod \"e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt\" (UID: \"068716fa-4615-4dbf-9c5f-41687865d5b1\") " pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:24:41.768989 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:41.768918 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/068716fa-4615-4dbf-9c5f-41687865d5b1-istio-data\") pod \"e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt\" (UID: \"068716fa-4615-4dbf-9c5f-41687865d5b1\") " pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:24:41.768989 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:41.768981 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/068716fa-4615-4dbf-9c5f-41687865d5b1-istio-podinfo\") pod \"e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt\" (UID: \"068716fa-4615-4dbf-9c5f-41687865d5b1\") " pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:24:41.769257 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:41.769013 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/068716fa-4615-4dbf-9c5f-41687865d5b1-istio-token\") pod \"e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt\" (UID: \"068716fa-4615-4dbf-9c5f-41687865d5b1\") " pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:24:41.769257 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:41.769064 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9btp\" (UniqueName: \"kubernetes.io/projected/068716fa-4615-4dbf-9c5f-41687865d5b1-kube-api-access-l9btp\") pod \"e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt\" (UID: \"068716fa-4615-4dbf-9c5f-41687865d5b1\") " pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:24:41.769257 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:41.769113 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/068716fa-4615-4dbf-9c5f-41687865d5b1-workload-certs\") pod \"e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt\" (UID: \"068716fa-4615-4dbf-9c5f-41687865d5b1\") " pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:24:41.769257 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:41.769153 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/068716fa-4615-4dbf-9c5f-41687865d5b1-credential-socket\") pod \"e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt\" (UID: \"068716fa-4615-4dbf-9c5f-41687865d5b1\") " pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:24:41.770113 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:41.769595 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/068716fa-4615-4dbf-9c5f-41687865d5b1-credential-socket\") pod \"e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt\" (UID: \"068716fa-4615-4dbf-9c5f-41687865d5b1\") " pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:24:41.770113 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:41.769655 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/068716fa-4615-4dbf-9c5f-41687865d5b1-istiod-ca-cert\") pod \"e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt\" (UID: \"068716fa-4615-4dbf-9c5f-41687865d5b1\") " pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:24:41.770113 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:41.769814 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/068716fa-4615-4dbf-9c5f-41687865d5b1-workload-certs\") pod \"e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt\" (UID: \"068716fa-4615-4dbf-9c5f-41687865d5b1\") " pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:24:41.770113 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:41.769865 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/068716fa-4615-4dbf-9c5f-41687865d5b1-workload-socket\") pod \"e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt\" (UID: \"068716fa-4615-4dbf-9c5f-41687865d5b1\") " pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:24:41.770113 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:41.770074 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/068716fa-4615-4dbf-9c5f-41687865d5b1-istio-data\") pod \"e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt\" (UID: \"068716fa-4615-4dbf-9c5f-41687865d5b1\") " pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:24:41.772209 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:41.772187 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/068716fa-4615-4dbf-9c5f-41687865d5b1-istio-envoy\") pod \"e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt\" (UID: \"068716fa-4615-4dbf-9c5f-41687865d5b1\") " pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:24:41.772318 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:41.772239 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/068716fa-4615-4dbf-9c5f-41687865d5b1-istio-podinfo\") pod \"e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt\" (UID: \"068716fa-4615-4dbf-9c5f-41687865d5b1\") " pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:24:41.831178 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:41.831143 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/068716fa-4615-4dbf-9c5f-41687865d5b1-istio-token\") pod \"e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt\" (UID: \"068716fa-4615-4dbf-9c5f-41687865d5b1\") " pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:24:41.831332 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:41.831223 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9btp\" (UniqueName: \"kubernetes.io/projected/068716fa-4615-4dbf-9c5f-41687865d5b1-kube-api-access-l9btp\") pod \"e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt\" (UID: \"068716fa-4615-4dbf-9c5f-41687865d5b1\") " pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:24:41.947935 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:41.947851 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:24:42.115117 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:42.115085 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:24:42.115635 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:42.115507 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:24:42.116261 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:42.116242 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:24:42.412198 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:24:42.412158 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod068716fa_4615_4dbf_9c5f_41687865d5b1.slice/crio-35fc601a2ab248de93b730888250e9bf7c0dc6e21e97f2d146b52dac21b23645 WatchSource:0}: Error finding container 35fc601a2ab248de93b730888250e9bf7c0dc6e21e97f2d146b52dac21b23645: Status 404 returned error can't find the container with id 35fc601a2ab248de93b730888250e9bf7c0dc6e21e97f2d146b52dac21b23645 Nov 01 06:24:42.413442 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:42.413418 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt"] Nov 01 06:24:42.414947 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:42.414911 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:24:42.415036 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:42.414980 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:24:42.415036 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:42.415008 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:24:42.731012 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:42.730915 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52" podUID="534945c2-d939-41cb-8125-dc2b4b788e40" containerName="main" containerID="cri-o://aedc9e828b59fd1df73dff29e8a217a438cc0d03ca2d5a579e97d3f58a7b215d" gracePeriod=45 Nov 01 06:24:43.001517 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:43.001472 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52" Nov 01 06:24:43.048175 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:43.048140 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8315d578-v4s9v"] Nov 01 06:24:43.051043 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:43.051019 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8315d578-v4s9v"] Nov 01 06:24:43.087380 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:43.087346 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/534945c2-d939-41cb-8125-dc2b4b788e40-home\" (UniqueName: \"kubernetes.io/empty-dir/534945c2-d939-41cb-8125-dc2b4b788e40-home\") pod \"534945c2-d939-41cb-8125-dc2b4b788e40\" (UID: \"534945c2-d939-41cb-8125-dc2b4b788e40\") " Nov 01 06:24:43.087580 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:43.087421 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/534945c2-d939-41cb-8125-dc2b4b788e40-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/534945c2-d939-41cb-8125-dc2b4b788e40-model-cache\") pod \"534945c2-d939-41cb-8125-dc2b4b788e40\" (UID: \"534945c2-d939-41cb-8125-dc2b4b788e40\") " Nov 01 06:24:43.087649 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:43.087612 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/534945c2-d939-41cb-8125-dc2b4b788e40-kube-api-access-f4wlk\" (UniqueName: \"kubernetes.io/projected/534945c2-d939-41cb-8125-dc2b4b788e40-kube-api-access-f4wlk\") pod \"534945c2-d939-41cb-8125-dc2b4b788e40\" (UID: \"534945c2-d939-41cb-8125-dc2b4b788e40\") " Nov 01 06:24:43.087723 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:43.087679 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/534945c2-d939-41cb-8125-dc2b4b788e40-dshm\" (UniqueName: \"kubernetes.io/empty-dir/534945c2-d939-41cb-8125-dc2b4b788e40-dshm\") pod \"534945c2-d939-41cb-8125-dc2b4b788e40\" (UID: \"534945c2-d939-41cb-8125-dc2b4b788e40\") " Nov 01 06:24:43.087775 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:43.087726 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/534945c2-d939-41cb-8125-dc2b4b788e40-home" pod "534945c2-d939-41cb-8125-dc2b4b788e40" (UID: "534945c2-d939-41cb-8125-dc2b4b788e40"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:24:43.087775 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:43.087730 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/534945c2-d939-41cb-8125-dc2b4b788e40-tls-certs\" (UniqueName: \"kubernetes.io/secret/534945c2-d939-41cb-8125-dc2b4b788e40-tls-certs\") pod \"534945c2-d939-41cb-8125-dc2b4b788e40\" (UID: \"534945c2-d939-41cb-8125-dc2b4b788e40\") " Nov 01 06:24:43.087860 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:43.087786 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/534945c2-d939-41cb-8125-dc2b4b788e40-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/534945c2-d939-41cb-8125-dc2b4b788e40-tmp-dir\") pod \"534945c2-d939-41cb-8125-dc2b4b788e40\" (UID: \"534945c2-d939-41cb-8125-dc2b4b788e40\") " Nov 01 06:24:43.087860 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:43.087742 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/534945c2-d939-41cb-8125-dc2b4b788e40-model-cache" pod "534945c2-d939-41cb-8125-dc2b4b788e40" (UID: "534945c2-d939-41cb-8125-dc2b4b788e40"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:24:43.088193 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:43.088141 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/534945c2-d939-41cb-8125-dc2b4b788e40-tmp-dir" pod "534945c2-d939-41cb-8125-dc2b4b788e40" (UID: "534945c2-d939-41cb-8125-dc2b4b788e40"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:24:43.088193 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:43.088182 5040 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/534945c2-d939-41cb-8125-dc2b4b788e40-model-cache\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:24:43.088385 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:43.088207 5040 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/534945c2-d939-41cb-8125-dc2b4b788e40-home\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:24:43.090044 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:43.090018 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/534945c2-d939-41cb-8125-dc2b4b788e40-dshm" pod "534945c2-d939-41cb-8125-dc2b4b788e40" (UID: "534945c2-d939-41cb-8125-dc2b4b788e40"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:24:43.090162 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:43.090134 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/534945c2-d939-41cb-8125-dc2b4b788e40-tls-certs" pod "534945c2-d939-41cb-8125-dc2b4b788e40" (UID: "534945c2-d939-41cb-8125-dc2b4b788e40"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:24:43.090243 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:43.090226 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/534945c2-d939-41cb-8125-dc2b4b788e40-kube-api-access-f4wlk" pod "534945c2-d939-41cb-8125-dc2b4b788e40" (UID: "534945c2-d939-41cb-8125-dc2b4b788e40"). InnerVolumeSpecName "kube-api-access-f4wlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:24:43.188905 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:43.188867 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4wlk\" (UniqueName: \"kubernetes.io/projected/534945c2-d939-41cb-8125-dc2b4b788e40-kube-api-access-f4wlk\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:24:43.188905 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:43.188899 5040 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/534945c2-d939-41cb-8125-dc2b4b788e40-dshm\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:24:43.188905 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:43.188910 5040 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/534945c2-d939-41cb-8125-dc2b4b788e40-tls-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:24:43.189366 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:43.188919 5040 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/534945c2-d939-41cb-8125-dc2b4b788e40-tmp-dir\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:24:43.362870 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:43.362827 5040 generic.go:356] "Generic (PLEG): container finished" podID="534945c2-d939-41cb-8125-dc2b4b788e40" containerID="aedc9e828b59fd1df73dff29e8a217a438cc0d03ca2d5a579e97d3f58a7b215d" exitCode=0 Nov 01 06:24:43.363106 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:43.363087 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52" event={"ID":"534945c2-d939-41cb-8125-dc2b4b788e40","Type":"ContainerDied","Data":"aedc9e828b59fd1df73dff29e8a217a438cc0d03ca2d5a579e97d3f58a7b215d"} Nov 01 06:24:43.363154 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:43.363132 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52" event={"ID":"534945c2-d939-41cb-8125-dc2b4b788e40","Type":"ContainerDied","Data":"b8f28e78d3f440cdcdc05a516c1d6e07a219603a2485701c1edc1f72f5cd3a15"} Nov 01 06:24:43.363189 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:43.363154 5040 scope.go:122] "RemoveContainer" containerID="aedc9e828b59fd1df73dff29e8a217a438cc0d03ca2d5a579e97d3f58a7b215d" Nov 01 06:24:43.363319 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:43.363307 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52" Nov 01 06:24:43.370988 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:43.370949 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" event={"ID":"068716fa-4615-4dbf-9c5f-41687865d5b1","Type":"ContainerStarted","Data":"44791eecd8e3223d742ac4c5c0b2f07057100ca4306ee7c0a39e839394079a15"} Nov 01 06:24:43.371146 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:43.371016 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" event={"ID":"068716fa-4615-4dbf-9c5f-41687865d5b1","Type":"ContainerStarted","Data":"35fc601a2ab248de93b730888250e9bf7c0dc6e21e97f2d146b52dac21b23645"} Nov 01 06:24:43.379391 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:43.379365 5040 scope.go:122] "RemoveContainer" containerID="aedc9e828b59fd1df73dff29e8a217a438cc0d03ca2d5a579e97d3f58a7b215d" Nov 01 06:24:43.379794 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:24:43.379765 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aedc9e828b59fd1df73dff29e8a217a438cc0d03ca2d5a579e97d3f58a7b215d\": container with ID starting with aedc9e828b59fd1df73dff29e8a217a438cc0d03ca2d5a579e97d3f58a7b215d not found: ID does not exist" containerID="aedc9e828b59fd1df73dff29e8a217a438cc0d03ca2d5a579e97d3f58a7b215d" Nov 01 06:24:43.380276 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:43.379808 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aedc9e828b59fd1df73dff29e8a217a438cc0d03ca2d5a579e97d3f58a7b215d"} err="failed to get container status \"aedc9e828b59fd1df73dff29e8a217a438cc0d03ca2d5a579e97d3f58a7b215d\": rpc error: code = NotFound desc = could not find container \"aedc9e828b59fd1df73dff29e8a217a438cc0d03ca2d5a579e97d3f58a7b215d\": container with ID starting with aedc9e828b59fd1df73dff29e8a217a438cc0d03ca2d5a579e97d3f58a7b215d not found: ID does not exist" Nov 01 06:24:43.514596 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:43.514563 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c42be41c-0e3d-41a1-be66-b2a8cf6f8940" path="/var/lib/kubelet/pods/c42be41c-0e3d-41a1-be66-b2a8cf6f8940/volumes" Nov 01 06:24:43.545657 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:43.545603 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" podStartSLOduration=2.545589644 podStartE2EDuration="2.545589644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:24:43.541802396 +0000 UTC m=+2362.196919563" watchObservedRunningTime="2026-11-01 06:24:43.545589644 +0000 UTC m=+2362.200706811" Nov 01 06:24:43.659924 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:43.659829 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52"] Nov 01 06:24:43.665167 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:43.665123 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-71cea4/auth-test-model-71cea4-kserve-5ccbfd498f-b4t52"] Nov 01 06:24:43.948136 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:43.948041 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:24:43.953997 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:43.953966 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:24:44.380311 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:44.380260 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:24:44.381235 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:44.381211 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:24:45.514723 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:45.514688 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="534945c2-d939-41cb-8125-dc2b4b788e40" path="/var/lib/kubelet/pods/534945c2-d939-41cb-8125-dc2b4b788e40/volumes" Nov 01 06:24:49.737404 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:49.737357 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4" podUID="3c295272-f68b-4568-8735-99890a86dd44" containerName="main" containerID="cri-o://d4ed5231833bef0218ce370bf31c4646860854fa1161b92e400824e0d71430f7" gracePeriod=45 Nov 01 06:24:49.782826 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:49.782787 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4" podUID="3c295272-f68b-4568-8735-99890a86dd44" containerName="main" probeResult="failure" output="Get \"https://10.132.0.151:8000/ready\": dial tcp 10.132.0.151:8000: connect: connection refused" Nov 01 06:24:50.002856 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:50.002830 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4" Nov 01 06:24:50.068517 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:50.068458 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c295272-f68b-4568-8735-99890a86dd44-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c295272-f68b-4568-8735-99890a86dd44-model-cache\") pod \"3c295272-f68b-4568-8735-99890a86dd44\" (UID: \"3c295272-f68b-4568-8735-99890a86dd44\") " Nov 01 06:24:50.068695 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:50.068566 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c295272-f68b-4568-8735-99890a86dd44-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c295272-f68b-4568-8735-99890a86dd44-dshm\") pod \"3c295272-f68b-4568-8735-99890a86dd44\" (UID: \"3c295272-f68b-4568-8735-99890a86dd44\") " Nov 01 06:24:50.068695 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:50.068603 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c295272-f68b-4568-8735-99890a86dd44-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c295272-f68b-4568-8735-99890a86dd44-tmp-dir\") pod \"3c295272-f68b-4568-8735-99890a86dd44\" (UID: \"3c295272-f68b-4568-8735-99890a86dd44\") " Nov 01 06:24:50.068818 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:50.068700 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c295272-f68b-4568-8735-99890a86dd44-kube-api-access-plhb7\" (UniqueName: \"kubernetes.io/projected/3c295272-f68b-4568-8735-99890a86dd44-kube-api-access-plhb7\") pod \"3c295272-f68b-4568-8735-99890a86dd44\" (UID: \"3c295272-f68b-4568-8735-99890a86dd44\") " Nov 01 06:24:50.068818 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:50.068731 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c295272-f68b-4568-8735-99890a86dd44-tls-certs\" (UniqueName: \"kubernetes.io/secret/3c295272-f68b-4568-8735-99890a86dd44-tls-certs\") pod \"3c295272-f68b-4568-8735-99890a86dd44\" (UID: \"3c295272-f68b-4568-8735-99890a86dd44\") " Nov 01 06:24:50.068818 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:50.068745 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c295272-f68b-4568-8735-99890a86dd44-model-cache" pod "3c295272-f68b-4568-8735-99890a86dd44" (UID: "3c295272-f68b-4568-8735-99890a86dd44"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:24:50.068818 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:50.068789 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c295272-f68b-4568-8735-99890a86dd44-home\" (UniqueName: \"kubernetes.io/empty-dir/3c295272-f68b-4568-8735-99890a86dd44-home\") pod \"3c295272-f68b-4568-8735-99890a86dd44\" (UID: \"3c295272-f68b-4568-8735-99890a86dd44\") " Nov 01 06:24:50.069005 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:50.068884 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c295272-f68b-4568-8735-99890a86dd44-tmp-dir" pod "3c295272-f68b-4568-8735-99890a86dd44" (UID: "3c295272-f68b-4568-8735-99890a86dd44"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:24:50.069112 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:50.069088 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c295272-f68b-4568-8735-99890a86dd44-home" pod "3c295272-f68b-4568-8735-99890a86dd44" (UID: "3c295272-f68b-4568-8735-99890a86dd44"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:24:50.069235 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:50.069217 5040 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c295272-f68b-4568-8735-99890a86dd44-model-cache\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:24:50.069291 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:50.069238 5040 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c295272-f68b-4568-8735-99890a86dd44-tmp-dir\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:24:50.069291 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:50.069249 5040 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c295272-f68b-4568-8735-99890a86dd44-home\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:24:50.070886 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:50.070831 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c295272-f68b-4568-8735-99890a86dd44-dshm" pod "3c295272-f68b-4568-8735-99890a86dd44" (UID: "3c295272-f68b-4568-8735-99890a86dd44"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:24:50.071017 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:50.070893 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c295272-f68b-4568-8735-99890a86dd44-tls-certs" pod "3c295272-f68b-4568-8735-99890a86dd44" (UID: "3c295272-f68b-4568-8735-99890a86dd44"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:24:50.071017 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:50.070942 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c295272-f68b-4568-8735-99890a86dd44-kube-api-access-plhb7" pod "3c295272-f68b-4568-8735-99890a86dd44" (UID: "3c295272-f68b-4568-8735-99890a86dd44"). InnerVolumeSpecName "kube-api-access-plhb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:24:50.170041 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:50.170013 5040 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c295272-f68b-4568-8735-99890a86dd44-dshm\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:24:50.170041 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:50.170040 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plhb7\" (UniqueName: \"kubernetes.io/projected/3c295272-f68b-4568-8735-99890a86dd44-kube-api-access-plhb7\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:24:50.170242 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:50.170050 5040 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c295272-f68b-4568-8735-99890a86dd44-tls-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:24:50.420752 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:50.420662 5040 generic.go:356] "Generic (PLEG): container finished" podID="3c295272-f68b-4568-8735-99890a86dd44" containerID="d4ed5231833bef0218ce370bf31c4646860854fa1161b92e400824e0d71430f7" exitCode=0 Nov 01 06:24:50.420752 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:50.420709 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4" Nov 01 06:24:50.420957 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:50.420746 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4" event={"ID":"3c295272-f68b-4568-8735-99890a86dd44","Type":"ContainerDied","Data":"d4ed5231833bef0218ce370bf31c4646860854fa1161b92e400824e0d71430f7"} Nov 01 06:24:50.420957 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:50.420801 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4" event={"ID":"3c295272-f68b-4568-8735-99890a86dd44","Type":"ContainerDied","Data":"c37940da6cb819d3925c9156f69746a16058ecee76e22a29476647d9a56aa292"} Nov 01 06:24:50.420957 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:50.420822 5040 scope.go:122] "RemoveContainer" containerID="d4ed5231833bef0218ce370bf31c4646860854fa1161b92e400824e0d71430f7" Nov 01 06:24:50.431439 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:50.431421 5040 scope.go:122] "RemoveContainer" containerID="d4ed5231833bef0218ce370bf31c4646860854fa1161b92e400824e0d71430f7" Nov 01 06:24:50.431723 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:24:50.431704 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4ed5231833bef0218ce370bf31c4646860854fa1161b92e400824e0d71430f7\": container with ID starting with d4ed5231833bef0218ce370bf31c4646860854fa1161b92e400824e0d71430f7 not found: ID does not exist" containerID="d4ed5231833bef0218ce370bf31c4646860854fa1161b92e400824e0d71430f7" Nov 01 06:24:50.431799 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:50.431730 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4ed5231833bef0218ce370bf31c4646860854fa1161b92e400824e0d71430f7"} err="failed to get container status \"d4ed5231833bef0218ce370bf31c4646860854fa1161b92e400824e0d71430f7\": rpc error: code = NotFound desc = could not find container \"d4ed5231833bef0218ce370bf31c4646860854fa1161b92e400824e0d71430f7\": container with ID starting with d4ed5231833bef0218ce370bf31c4646860854fa1161b92e400824e0d71430f7 not found: ID does not exist" Nov 01 06:24:50.587677 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:50.587643 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4"] Nov 01 06:24:50.593041 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:50.593010 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-99e79c/auth-test-model-99e79c-kserve-65fd6c4db9-lhkj4"] Nov 01 06:24:51.516271 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:24:51.516239 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c295272-f68b-4568-8735-99890a86dd44" path="/var/lib/kubelet/pods/3c295272-f68b-4568-8735-99890a86dd44/volumes" Nov 01 06:25:08.911826 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:08.911789 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a355f03b-qsd22"] Nov 01 06:25:08.912343 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:08.912295 5040 cpu_manager.go:414] "containerMap: removing container" podUID="534945c2-d939-41cb-8125-dc2b4b788e40" containerName="main" Nov 01 06:25:08.912343 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:08.912310 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="534945c2-d939-41cb-8125-dc2b4b788e40" containerName="main" Nov 01 06:25:08.912343 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:08.912326 5040 cpu_manager.go:414] "containerMap: removing container" podUID="3c295272-f68b-4568-8735-99890a86dd44" containerName="main" Nov 01 06:25:08.912343 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:08.912331 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c295272-f68b-4568-8735-99890a86dd44" containerName="main" Nov 01 06:25:08.912508 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:08.912439 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="534945c2-d939-41cb-8125-dc2b4b788e40" containerName="main" Nov 01 06:25:08.912508 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:08.912448 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c295272-f68b-4568-8735-99890a86dd44" containerName="main" Nov 01 06:25:08.917449 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:08.917430 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a355f03b-qsd22" Nov 01 06:25:08.965710 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:08.965676 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/799dab13-f58a-4400-ab1f-39a0b86d7e24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a355f03b-qsd22\" (UID: \"799dab13-f58a-4400-ab1f-39a0b86d7e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a355f03b-qsd22" Nov 01 06:25:08.965870 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:08.965734 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/799dab13-f58a-4400-ab1f-39a0b86d7e24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a355f03b-qsd22\" (UID: \"799dab13-f58a-4400-ab1f-39a0b86d7e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a355f03b-qsd22" Nov 01 06:25:08.965913 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:08.965862 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhbzr\" (UniqueName: \"kubernetes.io/projected/799dab13-f58a-4400-ab1f-39a0b86d7e24-kube-api-access-nhbzr\") pod \"maas-api-revoke-sub-a355f03b-qsd22\" (UID: \"799dab13-f58a-4400-ab1f-39a0b86d7e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a355f03b-qsd22" Nov 01 06:25:08.973209 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:08.973183 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:25:09.023789 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:09.023759 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:25:09.023955 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:09.023845 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:25:09.023955 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:09.023844 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:25:09.027061 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:09.027041 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a355f03b-qsd22"] Nov 01 06:25:09.066925 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:09.066897 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhbzr\" (UniqueName: \"kubernetes.io/projected/799dab13-f58a-4400-ab1f-39a0b86d7e24-kube-api-access-nhbzr\") pod \"maas-api-revoke-sub-a355f03b-qsd22\" (UID: \"799dab13-f58a-4400-ab1f-39a0b86d7e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a355f03b-qsd22" Nov 01 06:25:09.067068 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:09.066965 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/799dab13-f58a-4400-ab1f-39a0b86d7e24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a355f03b-qsd22\" (UID: \"799dab13-f58a-4400-ab1f-39a0b86d7e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a355f03b-qsd22" Nov 01 06:25:09.067068 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:09.067008 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/799dab13-f58a-4400-ab1f-39a0b86d7e24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a355f03b-qsd22\" (UID: \"799dab13-f58a-4400-ab1f-39a0b86d7e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a355f03b-qsd22" Nov 01 06:25:09.067692 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:09.067672 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/799dab13-f58a-4400-ab1f-39a0b86d7e24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a355f03b-qsd22\" (UID: \"799dab13-f58a-4400-ab1f-39a0b86d7e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a355f03b-qsd22" Nov 01 06:25:09.069339 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:09.069316 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/799dab13-f58a-4400-ab1f-39a0b86d7e24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a355f03b-qsd22\" (UID: \"799dab13-f58a-4400-ab1f-39a0b86d7e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a355f03b-qsd22" Nov 01 06:25:09.128143 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:09.128114 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhbzr\" (UniqueName: \"kubernetes.io/projected/799dab13-f58a-4400-ab1f-39a0b86d7e24-kube-api-access-nhbzr\") pod \"maas-api-revoke-sub-a355f03b-qsd22\" (UID: \"799dab13-f58a-4400-ab1f-39a0b86d7e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a355f03b-qsd22" Nov 01 06:25:09.229492 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:09.229298 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a355f03b-qsd22" Nov 01 06:25:09.474335 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:09.474296 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3f698a5-tcn6w"] Nov 01 06:25:09.478521 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:09.478502 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f698a5-tcn6w" Nov 01 06:25:09.536090 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:09.536058 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3f698a5-tcn6w"] Nov 01 06:25:09.573629 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:09.573589 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0526dc9-6d93-44f5-8f41-094310810816-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3f698a5-tcn6w\" (UID: \"a0526dc9-6d93-44f5-8f41-094310810816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f698a5-tcn6w" Nov 01 06:25:09.573828 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:09.573805 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn87w\" (UniqueName: \"kubernetes.io/projected/a0526dc9-6d93-44f5-8f41-094310810816-kube-api-access-kn87w\") pod \"maas-api-revoke-sub-a3f698a5-tcn6w\" (UID: \"a0526dc9-6d93-44f5-8f41-094310810816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f698a5-tcn6w" Nov 01 06:25:09.573870 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:09.573845 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0526dc9-6d93-44f5-8f41-094310810816-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3f698a5-tcn6w\" (UID: \"a0526dc9-6d93-44f5-8f41-094310810816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f698a5-tcn6w" Nov 01 06:25:09.663927 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:09.663896 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a355f03b-qsd22"] Nov 01 06:25:09.664118 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:25:09.664092 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod799dab13_f58a_4400_ab1f_39a0b86d7e24.slice/crio-33bc74df96b4198fb219d5d9a73388b27d7496be09f054e589cadd6b52a1d68f WatchSource:0}: Error finding container 33bc74df96b4198fb219d5d9a73388b27d7496be09f054e589cadd6b52a1d68f: Status 404 returned error can't find the container with id 33bc74df96b4198fb219d5d9a73388b27d7496be09f054e589cadd6b52a1d68f Nov 01 06:25:09.675358 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:09.675332 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0526dc9-6d93-44f5-8f41-094310810816-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3f698a5-tcn6w\" (UID: \"a0526dc9-6d93-44f5-8f41-094310810816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f698a5-tcn6w" Nov 01 06:25:09.675473 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:09.675458 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn87w\" (UniqueName: \"kubernetes.io/projected/a0526dc9-6d93-44f5-8f41-094310810816-kube-api-access-kn87w\") pod \"maas-api-revoke-sub-a3f698a5-tcn6w\" (UID: \"a0526dc9-6d93-44f5-8f41-094310810816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f698a5-tcn6w" Nov 01 06:25:09.675584 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:09.675513 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0526dc9-6d93-44f5-8f41-094310810816-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3f698a5-tcn6w\" (UID: \"a0526dc9-6d93-44f5-8f41-094310810816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f698a5-tcn6w" Nov 01 06:25:09.676176 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:09.676148 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0526dc9-6d93-44f5-8f41-094310810816-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3f698a5-tcn6w\" (UID: \"a0526dc9-6d93-44f5-8f41-094310810816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f698a5-tcn6w" Nov 01 06:25:09.677705 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:09.677682 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0526dc9-6d93-44f5-8f41-094310810816-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3f698a5-tcn6w\" (UID: \"a0526dc9-6d93-44f5-8f41-094310810816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f698a5-tcn6w" Nov 01 06:25:09.764441 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:09.764406 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn87w\" (UniqueName: \"kubernetes.io/projected/a0526dc9-6d93-44f5-8f41-094310810816-kube-api-access-kn87w\") pod \"maas-api-revoke-sub-a3f698a5-tcn6w\" (UID: \"a0526dc9-6d93-44f5-8f41-094310810816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f698a5-tcn6w" Nov 01 06:25:09.792410 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:09.792381 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f698a5-tcn6w" Nov 01 06:25:09.974873 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:25:09.974815 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:25:10.168683 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:10.168609 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3f698a5-tcn6w"] Nov 01 06:25:10.172906 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:25:10.172880 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0526dc9_6d93_44f5_8f41_094310810816.slice/crio-caef795491a6d9759e1b6c4b7d7ae079fd606b101623681b23a30ab8324ec083 WatchSource:0}: Error finding container caef795491a6d9759e1b6c4b7d7ae079fd606b101623681b23a30ab8324ec083: Status 404 returned error can't find the container with id caef795491a6d9759e1b6c4b7d7ae079fd606b101623681b23a30ab8324ec083 Nov 01 06:25:10.557270 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:10.557225 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a355f03b-qsd22" event={"ID":"799dab13-f58a-4400-ab1f-39a0b86d7e24","Type":"ContainerStarted","Data":"13268b14f96b56df34126da9bf1dfbf076aa77033cbdb4ccb35184edf7fbe6a8"} Nov 01 06:25:10.557471 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:10.557295 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a355f03b-qsd22" event={"ID":"799dab13-f58a-4400-ab1f-39a0b86d7e24","Type":"ContainerStarted","Data":"33bc74df96b4198fb219d5d9a73388b27d7496be09f054e589cadd6b52a1d68f"} Nov 01 06:25:10.558597 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:10.558570 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f698a5-tcn6w" event={"ID":"a0526dc9-6d93-44f5-8f41-094310810816","Type":"ContainerStarted","Data":"caef795491a6d9759e1b6c4b7d7ae079fd606b101623681b23a30ab8324ec083"} Nov 01 06:25:10.673981 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:10.673936 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a355f03b-qsd22" podStartSLOduration=2.673922433 podStartE2EDuration="2.673922433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:25:10.673505462 +0000 UTC m=+2389.328622623" watchObservedRunningTime="2026-11-01 06:25:10.673922433 +0000 UTC m=+2389.329039597" Nov 01 06:25:11.567503 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:11.567437 5040 generic.go:356] "Generic (PLEG): container finished" podID="799dab13-f58a-4400-ab1f-39a0b86d7e24" containerID="13268b14f96b56df34126da9bf1dfbf076aa77033cbdb4ccb35184edf7fbe6a8" exitCode=0 Nov 01 06:25:11.567964 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:11.567537 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a355f03b-qsd22" event={"ID":"799dab13-f58a-4400-ab1f-39a0b86d7e24","Type":"ContainerDied","Data":"13268b14f96b56df34126da9bf1dfbf076aa77033cbdb4ccb35184edf7fbe6a8"} Nov 01 06:25:11.569474 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:11.569452 5040 generic.go:356] "Generic (PLEG): container finished" podID="a0526dc9-6d93-44f5-8f41-094310810816" containerID="439f64f5f5c1c793062a51001281da16a2cf5a52d4bd9a5b76af9bb80ec4725d" exitCode=0 Nov 01 06:25:11.569613 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:11.569504 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f698a5-tcn6w" event={"ID":"a0526dc9-6d93-44f5-8f41-094310810816","Type":"ContainerDied","Data":"439f64f5f5c1c793062a51001281da16a2cf5a52d4bd9a5b76af9bb80ec4725d"} Nov 01 06:25:12.733162 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:12.733139 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a355f03b-qsd22" Nov 01 06:25:12.763379 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:12.763355 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f698a5-tcn6w" Nov 01 06:25:12.811643 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:12.811609 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0526dc9-6d93-44f5-8f41-094310810816-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0526dc9-6d93-44f5-8f41-094310810816-maas-api-xks-ca\") pod \"a0526dc9-6d93-44f5-8f41-094310810816\" (UID: \"a0526dc9-6d93-44f5-8f41-094310810816\") " Nov 01 06:25:12.811814 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:12.811685 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0526dc9-6d93-44f5-8f41-094310810816-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0526dc9-6d93-44f5-8f41-094310810816-maas-api-openshift-ca\") pod \"a0526dc9-6d93-44f5-8f41-094310810816\" (UID: \"a0526dc9-6d93-44f5-8f41-094310810816\") " Nov 01 06:25:12.811814 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:12.811775 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0526dc9-6d93-44f5-8f41-094310810816-kube-api-access-kn87w\" (UniqueName: \"kubernetes.io/projected/a0526dc9-6d93-44f5-8f41-094310810816-kube-api-access-kn87w\") pod \"a0526dc9-6d93-44f5-8f41-094310810816\" (UID: \"a0526dc9-6d93-44f5-8f41-094310810816\") " Nov 01 06:25:12.811938 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:12.811837 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/799dab13-f58a-4400-ab1f-39a0b86d7e24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/799dab13-f58a-4400-ab1f-39a0b86d7e24-maas-api-openshift-ca\") pod \"799dab13-f58a-4400-ab1f-39a0b86d7e24\" (UID: \"799dab13-f58a-4400-ab1f-39a0b86d7e24\") " Nov 01 06:25:12.811938 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:12.811871 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/799dab13-f58a-4400-ab1f-39a0b86d7e24-kube-api-access-nhbzr\" (UniqueName: \"kubernetes.io/projected/799dab13-f58a-4400-ab1f-39a0b86d7e24-kube-api-access-nhbzr\") pod \"799dab13-f58a-4400-ab1f-39a0b86d7e24\" (UID: \"799dab13-f58a-4400-ab1f-39a0b86d7e24\") " Nov 01 06:25:12.811938 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:12.811908 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/799dab13-f58a-4400-ab1f-39a0b86d7e24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/799dab13-f58a-4400-ab1f-39a0b86d7e24-maas-api-xks-ca\") pod \"799dab13-f58a-4400-ab1f-39a0b86d7e24\" (UID: \"799dab13-f58a-4400-ab1f-39a0b86d7e24\") " Nov 01 06:25:12.812086 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:12.812000 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0526dc9-6d93-44f5-8f41-094310810816-maas-api-openshift-ca" pod "a0526dc9-6d93-44f5-8f41-094310810816" (UID: "a0526dc9-6d93-44f5-8f41-094310810816"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:25:12.812243 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:12.812207 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/799dab13-f58a-4400-ab1f-39a0b86d7e24-maas-api-openshift-ca" pod "799dab13-f58a-4400-ab1f-39a0b86d7e24" (UID: "799dab13-f58a-4400-ab1f-39a0b86d7e24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:25:12.812557 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:12.812534 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0526dc9-6d93-44f5-8f41-094310810816-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:12.812661 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:12.812562 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/799dab13-f58a-4400-ab1f-39a0b86d7e24-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:12.813845 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:12.813823 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0526dc9-6d93-44f5-8f41-094310810816-maas-api-xks-ca" pod "a0526dc9-6d93-44f5-8f41-094310810816" (UID: "a0526dc9-6d93-44f5-8f41-094310810816"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:25:12.814011 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:12.813991 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/799dab13-f58a-4400-ab1f-39a0b86d7e24-kube-api-access-nhbzr" pod "799dab13-f58a-4400-ab1f-39a0b86d7e24" (UID: "799dab13-f58a-4400-ab1f-39a0b86d7e24"). InnerVolumeSpecName "kube-api-access-nhbzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:25:12.814073 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:12.814020 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0526dc9-6d93-44f5-8f41-094310810816-kube-api-access-kn87w" pod "a0526dc9-6d93-44f5-8f41-094310810816" (UID: "a0526dc9-6d93-44f5-8f41-094310810816"). InnerVolumeSpecName "kube-api-access-kn87w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:25:12.814073 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:12.814019 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/799dab13-f58a-4400-ab1f-39a0b86d7e24-maas-api-xks-ca" pod "799dab13-f58a-4400-ab1f-39a0b86d7e24" (UID: "799dab13-f58a-4400-ab1f-39a0b86d7e24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:25:12.913986 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:12.913894 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0526dc9-6d93-44f5-8f41-094310810816-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:12.913986 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:12.913927 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn87w\" (UniqueName: \"kubernetes.io/projected/a0526dc9-6d93-44f5-8f41-094310810816-kube-api-access-kn87w\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:12.913986 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:12.913942 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhbzr\" (UniqueName: \"kubernetes.io/projected/799dab13-f58a-4400-ab1f-39a0b86d7e24-kube-api-access-nhbzr\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:12.913986 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:12.913954 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/799dab13-f58a-4400-ab1f-39a0b86d7e24-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:13.584004 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:13.583974 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a355f03b-qsd22" Nov 01 06:25:13.584205 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:13.584003 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a355f03b-qsd22" event={"ID":"799dab13-f58a-4400-ab1f-39a0b86d7e24","Type":"ContainerDied","Data":"33bc74df96b4198fb219d5d9a73388b27d7496be09f054e589cadd6b52a1d68f"} Nov 01 06:25:13.584205 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:13.584056 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33bc74df96b4198fb219d5d9a73388b27d7496be09f054e589cadd6b52a1d68f" Nov 01 06:25:13.585520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:13.585469 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f698a5-tcn6w" event={"ID":"a0526dc9-6d93-44f5-8f41-094310810816","Type":"ContainerDied","Data":"caef795491a6d9759e1b6c4b7d7ae079fd606b101623681b23a30ab8324ec083"} Nov 01 06:25:13.585651 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:13.585524 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="caef795491a6d9759e1b6c4b7d7ae079fd606b101623681b23a30ab8324ec083" Nov 01 06:25:13.585651 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:13.585529 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f698a5-tcn6w" Nov 01 06:25:20.534561 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:20.534523 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfca6de2-vdpw6"] Nov 01 06:25:20.534987 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:20.534973 5040 cpu_manager.go:414] "containerMap: removing container" podUID="799dab13-f58a-4400-ab1f-39a0b86d7e24" containerName="revoke-keys" Nov 01 06:25:20.534987 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:20.534986 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="799dab13-f58a-4400-ab1f-39a0b86d7e24" containerName="revoke-keys" Nov 01 06:25:20.535070 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:20.535009 5040 cpu_manager.go:414] "containerMap: removing container" podUID="a0526dc9-6d93-44f5-8f41-094310810816" containerName="revoke-keys" Nov 01 06:25:20.535070 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:20.535014 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0526dc9-6d93-44f5-8f41-094310810816" containerName="revoke-keys" Nov 01 06:25:20.535140 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:20.535121 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0526dc9-6d93-44f5-8f41-094310810816" containerName="revoke-keys" Nov 01 06:25:20.535140 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:20.535136 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="799dab13-f58a-4400-ab1f-39a0b86d7e24" containerName="revoke-keys" Nov 01 06:25:20.538319 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:20.538302 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfca6de2-vdpw6" Nov 01 06:25:20.592752 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:20.592722 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:25:20.598638 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:20.598599 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c45608df-359e-41fb-90c7-917b59e9042c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfca6de2-vdpw6\" (UID: \"c45608df-359e-41fb-90c7-917b59e9042c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfca6de2-vdpw6" Nov 01 06:25:20.598778 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:20.598748 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c45608df-359e-41fb-90c7-917b59e9042c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfca6de2-vdpw6\" (UID: \"c45608df-359e-41fb-90c7-917b59e9042c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfca6de2-vdpw6" Nov 01 06:25:20.598853 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:20.598835 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkb2h\" (UniqueName: \"kubernetes.io/projected/c45608df-359e-41fb-90c7-917b59e9042c-kube-api-access-bkb2h\") pod \"maas-api-revoke-sub-dfca6de2-vdpw6\" (UID: \"c45608df-359e-41fb-90c7-917b59e9042c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfca6de2-vdpw6" Nov 01 06:25:20.643236 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:20.643202 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:25:20.643236 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:20.643237 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:25:20.643460 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:20.643372 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:25:20.646624 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:20.646603 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfca6de2-vdpw6"] Nov 01 06:25:20.700374 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:20.700338 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c45608df-359e-41fb-90c7-917b59e9042c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfca6de2-vdpw6\" (UID: \"c45608df-359e-41fb-90c7-917b59e9042c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfca6de2-vdpw6" Nov 01 06:25:20.700584 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:20.700405 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c45608df-359e-41fb-90c7-917b59e9042c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfca6de2-vdpw6\" (UID: \"c45608df-359e-41fb-90c7-917b59e9042c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfca6de2-vdpw6" Nov 01 06:25:20.700584 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:20.700451 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkb2h\" (UniqueName: \"kubernetes.io/projected/c45608df-359e-41fb-90c7-917b59e9042c-kube-api-access-bkb2h\") pod \"maas-api-revoke-sub-dfca6de2-vdpw6\" (UID: \"c45608df-359e-41fb-90c7-917b59e9042c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfca6de2-vdpw6" Nov 01 06:25:20.701126 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:20.701103 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c45608df-359e-41fb-90c7-917b59e9042c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfca6de2-vdpw6\" (UID: \"c45608df-359e-41fb-90c7-917b59e9042c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfca6de2-vdpw6" Nov 01 06:25:20.702715 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:20.702695 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c45608df-359e-41fb-90c7-917b59e9042c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfca6de2-vdpw6\" (UID: \"c45608df-359e-41fb-90c7-917b59e9042c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfca6de2-vdpw6" Nov 01 06:25:20.761790 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:20.761754 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkb2h\" (UniqueName: \"kubernetes.io/projected/c45608df-359e-41fb-90c7-917b59e9042c-kube-api-access-bkb2h\") pod \"maas-api-revoke-sub-dfca6de2-vdpw6\" (UID: \"c45608df-359e-41fb-90c7-917b59e9042c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfca6de2-vdpw6" Nov 01 06:25:20.851360 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:20.851273 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfca6de2-vdpw6" Nov 01 06:25:21.226513 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:21.226470 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfca6de2-vdpw6"] Nov 01 06:25:21.227106 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:25:21.227077 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc45608df_359e_41fb_90c7_917b59e9042c.slice/crio-218aa437ab916499257afc40ab1761b7437ff534704528ba14bdf2639693bfd0 WatchSource:0}: Error finding container 218aa437ab916499257afc40ab1761b7437ff534704528ba14bdf2639693bfd0: Status 404 returned error can't find the container with id 218aa437ab916499257afc40ab1761b7437ff534704528ba14bdf2639693bfd0 Nov 01 06:25:21.640373 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:21.640340 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfca6de2-vdpw6" event={"ID":"c45608df-359e-41fb-90c7-917b59e9042c","Type":"ContainerStarted","Data":"d9ce6f2cd7dc74f36882d98c9d340cfe3b3f0625b057fcbcb7e3f3dcd7d91ca7"} Nov 01 06:25:21.640765 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:21.640396 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfca6de2-vdpw6" event={"ID":"c45608df-359e-41fb-90c7-917b59e9042c","Type":"ContainerStarted","Data":"218aa437ab916499257afc40ab1761b7437ff534704528ba14bdf2639693bfd0"} Nov 01 06:25:21.758510 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:21.758383 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfca6de2-vdpw6" podStartSLOduration=1.758365923 podStartE2EDuration="1.758365923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:25:21.757255131 +0000 UTC m=+2400.412372299" watchObservedRunningTime="2026-11-01 06:25:21.758365923 +0000 UTC m=+2400.413483091" Nov 01 06:25:21.954549 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:21.954514 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 06:25:21.959191 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:21.959168 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 06:25:22.500630 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:22.500597 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddab543d-xn7hm"] Nov 01 06:25:22.504781 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:22.504759 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddab543d-xn7hm" Nov 01 06:25:22.561282 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:22.561245 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddab543d-xn7hm"] Nov 01 06:25:22.633871 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:22.633830 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jsj7\" (UniqueName: \"kubernetes.io/projected/0eec1ba8-ec11-49df-b49f-8beab1f82be5-kube-api-access-9jsj7\") pod \"maas-api-revoke-sub-ddab543d-xn7hm\" (UID: \"0eec1ba8-ec11-49df-b49f-8beab1f82be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddab543d-xn7hm" Nov 01 06:25:22.634029 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:22.633968 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eec1ba8-ec11-49df-b49f-8beab1f82be5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddab543d-xn7hm\" (UID: \"0eec1ba8-ec11-49df-b49f-8beab1f82be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddab543d-xn7hm" Nov 01 06:25:22.634087 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:22.634050 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eec1ba8-ec11-49df-b49f-8beab1f82be5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddab543d-xn7hm\" (UID: \"0eec1ba8-ec11-49df-b49f-8beab1f82be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddab543d-xn7hm" Nov 01 06:25:22.647958 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:22.647919 5040 generic.go:356] "Generic (PLEG): container finished" podID="c45608df-359e-41fb-90c7-917b59e9042c" containerID="d9ce6f2cd7dc74f36882d98c9d340cfe3b3f0625b057fcbcb7e3f3dcd7d91ca7" exitCode=0 Nov 01 06:25:22.648341 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:22.647986 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfca6de2-vdpw6" event={"ID":"c45608df-359e-41fb-90c7-917b59e9042c","Type":"ContainerDied","Data":"d9ce6f2cd7dc74f36882d98c9d340cfe3b3f0625b057fcbcb7e3f3dcd7d91ca7"} Nov 01 06:25:22.734965 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:22.734929 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eec1ba8-ec11-49df-b49f-8beab1f82be5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddab543d-xn7hm\" (UID: \"0eec1ba8-ec11-49df-b49f-8beab1f82be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddab543d-xn7hm" Nov 01 06:25:22.735148 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:22.735013 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eec1ba8-ec11-49df-b49f-8beab1f82be5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddab543d-xn7hm\" (UID: \"0eec1ba8-ec11-49df-b49f-8beab1f82be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddab543d-xn7hm" Nov 01 06:25:22.735148 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:22.735075 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jsj7\" (UniqueName: \"kubernetes.io/projected/0eec1ba8-ec11-49df-b49f-8beab1f82be5-kube-api-access-9jsj7\") pod \"maas-api-revoke-sub-ddab543d-xn7hm\" (UID: \"0eec1ba8-ec11-49df-b49f-8beab1f82be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddab543d-xn7hm" Nov 01 06:25:22.735666 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:22.735642 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eec1ba8-ec11-49df-b49f-8beab1f82be5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddab543d-xn7hm\" (UID: \"0eec1ba8-ec11-49df-b49f-8beab1f82be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddab543d-xn7hm" Nov 01 06:25:22.737326 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:22.737303 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eec1ba8-ec11-49df-b49f-8beab1f82be5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddab543d-xn7hm\" (UID: \"0eec1ba8-ec11-49df-b49f-8beab1f82be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddab543d-xn7hm" Nov 01 06:25:22.797142 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:22.797063 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jsj7\" (UniqueName: \"kubernetes.io/projected/0eec1ba8-ec11-49df-b49f-8beab1f82be5-kube-api-access-9jsj7\") pod \"maas-api-revoke-sub-ddab543d-xn7hm\" (UID: \"0eec1ba8-ec11-49df-b49f-8beab1f82be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddab543d-xn7hm" Nov 01 06:25:22.820325 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:22.820292 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddab543d-xn7hm" Nov 01 06:25:23.202841 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:23.202811 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddab543d-xn7hm"] Nov 01 06:25:23.203925 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:25:23.203899 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eec1ba8_ec11_49df_b49f_8beab1f82be5.slice/crio-ff1db764954c7f65453fa4e6464954b20d0b56e30c1f5d50b89d86a0938527f9 WatchSource:0}: Error finding container ff1db764954c7f65453fa4e6464954b20d0b56e30c1f5d50b89d86a0938527f9: Status 404 returned error can't find the container with id ff1db764954c7f65453fa4e6464954b20d0b56e30c1f5d50b89d86a0938527f9 Nov 01 06:25:23.656370 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:23.656333 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddab543d-xn7hm" event={"ID":"0eec1ba8-ec11-49df-b49f-8beab1f82be5","Type":"ContainerStarted","Data":"461836d3e61393cbe7bb9e76ee1770b28d4c570f34509590e092bd9cdc7b25ce"} Nov 01 06:25:23.656848 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:23.656387 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddab543d-xn7hm" event={"ID":"0eec1ba8-ec11-49df-b49f-8beab1f82be5","Type":"ContainerStarted","Data":"ff1db764954c7f65453fa4e6464954b20d0b56e30c1f5d50b89d86a0938527f9"} Nov 01 06:25:23.778357 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:23.778245 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddab543d-xn7hm" podStartSLOduration=1.778228072 podStartE2EDuration="1.778228072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:25:23.775770069 +0000 UTC m=+2402.430887242" watchObservedRunningTime="2026-11-01 06:25:23.778228072 +0000 UTC m=+2402.433345240" Nov 01 06:25:23.813001 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:23.812975 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfca6de2-vdpw6" Nov 01 06:25:23.966129 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:23.966093 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c45608df-359e-41fb-90c7-917b59e9042c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c45608df-359e-41fb-90c7-917b59e9042c-maas-api-xks-ca\") pod \"c45608df-359e-41fb-90c7-917b59e9042c\" (UID: \"c45608df-359e-41fb-90c7-917b59e9042c\") " Nov 01 06:25:23.966297 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:23.966167 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c45608df-359e-41fb-90c7-917b59e9042c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c45608df-359e-41fb-90c7-917b59e9042c-maas-api-openshift-ca\") pod \"c45608df-359e-41fb-90c7-917b59e9042c\" (UID: \"c45608df-359e-41fb-90c7-917b59e9042c\") " Nov 01 06:25:23.966297 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:23.966220 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c45608df-359e-41fb-90c7-917b59e9042c-kube-api-access-bkb2h\" (UniqueName: \"kubernetes.io/projected/c45608df-359e-41fb-90c7-917b59e9042c-kube-api-access-bkb2h\") pod \"c45608df-359e-41fb-90c7-917b59e9042c\" (UID: \"c45608df-359e-41fb-90c7-917b59e9042c\") " Nov 01 06:25:23.966564 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:23.966541 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c45608df-359e-41fb-90c7-917b59e9042c-maas-api-openshift-ca" pod "c45608df-359e-41fb-90c7-917b59e9042c" (UID: "c45608df-359e-41fb-90c7-917b59e9042c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:25:23.968323 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:23.968296 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c45608df-359e-41fb-90c7-917b59e9042c-maas-api-xks-ca" pod "c45608df-359e-41fb-90c7-917b59e9042c" (UID: "c45608df-359e-41fb-90c7-917b59e9042c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:25:23.968323 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:23.968299 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c45608df-359e-41fb-90c7-917b59e9042c-kube-api-access-bkb2h" pod "c45608df-359e-41fb-90c7-917b59e9042c" (UID: "c45608df-359e-41fb-90c7-917b59e9042c"). InnerVolumeSpecName "kube-api-access-bkb2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:25:24.067122 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:24.067093 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkb2h\" (UniqueName: \"kubernetes.io/projected/c45608df-359e-41fb-90c7-917b59e9042c-kube-api-access-bkb2h\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:24.067249 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:24.067125 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c45608df-359e-41fb-90c7-917b59e9042c-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:24.067249 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:24.067139 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c45608df-359e-41fb-90c7-917b59e9042c-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:24.663907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:24.663875 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfca6de2-vdpw6" Nov 01 06:25:24.664371 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:24.663867 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfca6de2-vdpw6" event={"ID":"c45608df-359e-41fb-90c7-917b59e9042c","Type":"ContainerDied","Data":"218aa437ab916499257afc40ab1761b7437ff534704528ba14bdf2639693bfd0"} Nov 01 06:25:24.664371 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:24.664018 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="218aa437ab916499257afc40ab1761b7437ff534704528ba14bdf2639693bfd0" Nov 01 06:25:24.665646 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:24.665621 5040 generic.go:356] "Generic (PLEG): container finished" podID="0eec1ba8-ec11-49df-b49f-8beab1f82be5" containerID="461836d3e61393cbe7bb9e76ee1770b28d4c570f34509590e092bd9cdc7b25ce" exitCode=0 Nov 01 06:25:24.665806 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:24.665656 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddab543d-xn7hm" event={"ID":"0eec1ba8-ec11-49df-b49f-8beab1f82be5","Type":"ContainerDied","Data":"461836d3e61393cbe7bb9e76ee1770b28d4c570f34509590e092bd9cdc7b25ce"} Nov 01 06:25:25.823963 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:25.823937 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddab543d-xn7hm" Nov 01 06:25:25.987353 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:25.987265 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eec1ba8-ec11-49df-b49f-8beab1f82be5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eec1ba8-ec11-49df-b49f-8beab1f82be5-maas-api-openshift-ca\") pod \"0eec1ba8-ec11-49df-b49f-8beab1f82be5\" (UID: \"0eec1ba8-ec11-49df-b49f-8beab1f82be5\") " Nov 01 06:25:25.987544 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:25.987368 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eec1ba8-ec11-49df-b49f-8beab1f82be5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eec1ba8-ec11-49df-b49f-8beab1f82be5-maas-api-xks-ca\") pod \"0eec1ba8-ec11-49df-b49f-8beab1f82be5\" (UID: \"0eec1ba8-ec11-49df-b49f-8beab1f82be5\") " Nov 01 06:25:25.987544 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:25.987403 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eec1ba8-ec11-49df-b49f-8beab1f82be5-kube-api-access-9jsj7\" (UniqueName: \"kubernetes.io/projected/0eec1ba8-ec11-49df-b49f-8beab1f82be5-kube-api-access-9jsj7\") pod \"0eec1ba8-ec11-49df-b49f-8beab1f82be5\" (UID: \"0eec1ba8-ec11-49df-b49f-8beab1f82be5\") " Nov 01 06:25:25.987733 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:25.987706 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eec1ba8-ec11-49df-b49f-8beab1f82be5-maas-api-openshift-ca" pod "0eec1ba8-ec11-49df-b49f-8beab1f82be5" (UID: "0eec1ba8-ec11-49df-b49f-8beab1f82be5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:25:25.989489 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:25.989450 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eec1ba8-ec11-49df-b49f-8beab1f82be5-maas-api-xks-ca" pod "0eec1ba8-ec11-49df-b49f-8beab1f82be5" (UID: "0eec1ba8-ec11-49df-b49f-8beab1f82be5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:25:25.989577 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:25.989556 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eec1ba8-ec11-49df-b49f-8beab1f82be5-kube-api-access-9jsj7" pod "0eec1ba8-ec11-49df-b49f-8beab1f82be5" (UID: "0eec1ba8-ec11-49df-b49f-8beab1f82be5"). InnerVolumeSpecName "kube-api-access-9jsj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:25:26.088980 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:26.088943 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eec1ba8-ec11-49df-b49f-8beab1f82be5-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:26.088980 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:26.088976 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jsj7\" (UniqueName: \"kubernetes.io/projected/0eec1ba8-ec11-49df-b49f-8beab1f82be5-kube-api-access-9jsj7\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:26.089192 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:26.088992 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eec1ba8-ec11-49df-b49f-8beab1f82be5-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:26.679800 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:26.679768 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddab543d-xn7hm" Nov 01 06:25:26.679979 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:26.679799 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddab543d-xn7hm" event={"ID":"0eec1ba8-ec11-49df-b49f-8beab1f82be5","Type":"ContainerDied","Data":"ff1db764954c7f65453fa4e6464954b20d0b56e30c1f5d50b89d86a0938527f9"} Nov 01 06:25:26.679979 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:26.679849 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff1db764954c7f65453fa4e6464954b20d0b56e30c1f5d50b89d86a0938527f9" Nov 01 06:25:31.953627 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:31.953592 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e40-6t258"] Nov 01 06:25:31.954083 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:31.954069 5040 cpu_manager.go:414] "containerMap: removing container" podUID="0eec1ba8-ec11-49df-b49f-8beab1f82be5" containerName="revoke-keys" Nov 01 06:25:31.954128 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:31.954084 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eec1ba8-ec11-49df-b49f-8beab1f82be5" containerName="revoke-keys" Nov 01 06:25:31.954128 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:31.954112 5040 cpu_manager.go:414] "containerMap: removing container" podUID="c45608df-359e-41fb-90c7-917b59e9042c" containerName="revoke-keys" Nov 01 06:25:31.954128 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:31.954118 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c45608df-359e-41fb-90c7-917b59e9042c" containerName="revoke-keys" Nov 01 06:25:31.954212 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:31.954204 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eec1ba8-ec11-49df-b49f-8beab1f82be5" containerName="revoke-keys" Nov 01 06:25:31.954243 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:31.954217 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="c45608df-359e-41fb-90c7-917b59e9042c" containerName="revoke-keys" Nov 01 06:25:31.957671 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:31.957654 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e40-6t258" Nov 01 06:25:32.050254 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:32.050211 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fda7e954-48ee-4e6c-b04a-71df9940cb5a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-83558e40-6t258\" (UID: \"fda7e954-48ee-4e6c-b04a-71df9940cb5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e40-6t258" Nov 01 06:25:32.050463 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:32.050346 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fda7e954-48ee-4e6c-b04a-71df9940cb5a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-83558e40-6t258\" (UID: \"fda7e954-48ee-4e6c-b04a-71df9940cb5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e40-6t258" Nov 01 06:25:32.050463 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:32.050447 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw6bd\" (UniqueName: \"kubernetes.io/projected/fda7e954-48ee-4e6c-b04a-71df9940cb5a-kube-api-access-pw6bd\") pod \"maas-api-revoke-keys-e2e-ait-83558e40-6t258\" (UID: \"fda7e954-48ee-4e6c-b04a-71df9940cb5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e40-6t258" Nov 01 06:25:32.060974 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:32.060935 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:25:32.061144 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:32.061086 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:25:32.061144 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:32.061118 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:25:32.061254 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:32.061157 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:25:32.064598 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:32.064578 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e40-6t258"] Nov 01 06:25:32.151357 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:32.151323 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fda7e954-48ee-4e6c-b04a-71df9940cb5a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-83558e40-6t258\" (UID: \"fda7e954-48ee-4e6c-b04a-71df9940cb5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e40-6t258" Nov 01 06:25:32.151561 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:32.151414 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw6bd\" (UniqueName: \"kubernetes.io/projected/fda7e954-48ee-4e6c-b04a-71df9940cb5a-kube-api-access-pw6bd\") pod \"maas-api-revoke-keys-e2e-ait-83558e40-6t258\" (UID: \"fda7e954-48ee-4e6c-b04a-71df9940cb5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e40-6t258" Nov 01 06:25:32.151621 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:32.151608 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fda7e954-48ee-4e6c-b04a-71df9940cb5a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-83558e40-6t258\" (UID: \"fda7e954-48ee-4e6c-b04a-71df9940cb5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e40-6t258" Nov 01 06:25:32.152260 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:32.152232 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fda7e954-48ee-4e6c-b04a-71df9940cb5a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-83558e40-6t258\" (UID: \"fda7e954-48ee-4e6c-b04a-71df9940cb5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e40-6t258" Nov 01 06:25:32.153886 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:32.153861 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fda7e954-48ee-4e6c-b04a-71df9940cb5a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-83558e40-6t258\" (UID: \"fda7e954-48ee-4e6c-b04a-71df9940cb5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e40-6t258" Nov 01 06:25:32.213879 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:32.213806 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw6bd\" (UniqueName: \"kubernetes.io/projected/fda7e954-48ee-4e6c-b04a-71df9940cb5a-kube-api-access-pw6bd\") pod \"maas-api-revoke-keys-e2e-ait-83558e40-6t258\" (UID: \"fda7e954-48ee-4e6c-b04a-71df9940cb5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e40-6t258" Nov 01 06:25:32.271120 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:32.271085 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e40-6t258" Nov 01 06:25:32.650194 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:32.650153 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e40-6t258"] Nov 01 06:25:32.653740 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:25:32.653701 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda7e954_48ee_4e6c_b04a_71df9940cb5a.slice/crio-a1e2873033cb187eb73f77eba95bf3ba4c4b9a3257f4f33c7747cc7a306267cf WatchSource:0}: Error finding container a1e2873033cb187eb73f77eba95bf3ba4c4b9a3257f4f33c7747cc7a306267cf: Status 404 returned error can't find the container with id a1e2873033cb187eb73f77eba95bf3ba4c4b9a3257f4f33c7747cc7a306267cf Nov 01 06:25:32.719652 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:32.719617 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e40-6t258" event={"ID":"fda7e954-48ee-4e6c-b04a-71df9940cb5a","Type":"ContainerStarted","Data":"a1e2873033cb187eb73f77eba95bf3ba4c4b9a3257f4f33c7747cc7a306267cf"} Nov 01 06:25:33.732323 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:33.732220 5040 generic.go:356] "Generic (PLEG): container finished" podID="fda7e954-48ee-4e6c-b04a-71df9940cb5a" containerID="2519dcc32bbaed2948fa886551cd783ac8b0158fb71a8e13ebb02b584972a952" exitCode=0 Nov 01 06:25:33.732323 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:33.732304 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e40-6t258" event={"ID":"fda7e954-48ee-4e6c-b04a-71df9940cb5a","Type":"ContainerDied","Data":"2519dcc32bbaed2948fa886551cd783ac8b0158fb71a8e13ebb02b584972a952"} Nov 01 06:25:34.377146 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:34.377112 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-cd78f1/e2e-usage-sim-54291c-7c768bf864-pjszd"] Nov 01 06:25:34.377325 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:34.377300 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-cd78f1/e2e-usage-sim-54291c-7c768bf864-pjszd" podUID="6e0b1e48-0511-4ef7-8f73-1106983b5b33" containerName="sim" containerID="cri-o://dbf32ba40689e755ac7a70b3822931f131352e80384eb04c5f62191003d74273" gracePeriod=30 Nov 01 06:25:34.658177 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:34.658155 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-cd78f1/e2e-usage-sim-54291c-7c768bf864-pjszd" Nov 01 06:25:34.676077 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:34.676050 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e0b1e48-0511-4ef7-8f73-1106983b5b33-kube-api-access-lzkd7\" (UniqueName: \"kubernetes.io/projected/6e0b1e48-0511-4ef7-8f73-1106983b5b33-kube-api-access-lzkd7\") pod \"6e0b1e48-0511-4ef7-8f73-1106983b5b33\" (UID: \"6e0b1e48-0511-4ef7-8f73-1106983b5b33\") " Nov 01 06:25:34.676217 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:34.676112 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e0b1e48-0511-4ef7-8f73-1106983b5b33-tls\" (UniqueName: \"kubernetes.io/secret/6e0b1e48-0511-4ef7-8f73-1106983b5b33-tls\") pod \"6e0b1e48-0511-4ef7-8f73-1106983b5b33\" (UID: \"6e0b1e48-0511-4ef7-8f73-1106983b5b33\") " Nov 01 06:25:34.678327 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:34.678298 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e0b1e48-0511-4ef7-8f73-1106983b5b33-kube-api-access-lzkd7" pod "6e0b1e48-0511-4ef7-8f73-1106983b5b33" (UID: "6e0b1e48-0511-4ef7-8f73-1106983b5b33"). InnerVolumeSpecName "kube-api-access-lzkd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:25:34.678327 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:34.678316 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e0b1e48-0511-4ef7-8f73-1106983b5b33-tls" pod "6e0b1e48-0511-4ef7-8f73-1106983b5b33" (UID: "6e0b1e48-0511-4ef7-8f73-1106983b5b33"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:25:34.739947 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:34.739913 5040 generic.go:356] "Generic (PLEG): container finished" podID="6e0b1e48-0511-4ef7-8f73-1106983b5b33" containerID="dbf32ba40689e755ac7a70b3822931f131352e80384eb04c5f62191003d74273" exitCode=0 Nov 01 06:25:34.740357 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:34.739954 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-cd78f1/e2e-usage-sim-54291c-7c768bf864-pjszd" Nov 01 06:25:34.740357 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:34.739987 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-cd78f1/e2e-usage-sim-54291c-7c768bf864-pjszd" event={"ID":"6e0b1e48-0511-4ef7-8f73-1106983b5b33","Type":"ContainerDied","Data":"dbf32ba40689e755ac7a70b3822931f131352e80384eb04c5f62191003d74273"} Nov 01 06:25:34.740357 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:34.740051 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-cd78f1/e2e-usage-sim-54291c-7c768bf864-pjszd" event={"ID":"6e0b1e48-0511-4ef7-8f73-1106983b5b33","Type":"ContainerDied","Data":"d5c0b5566c523f1455424b834800a788c0790b9732ea7e694e2a548e33a8daf1"} Nov 01 06:25:34.740357 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:34.740070 5040 scope.go:122] "RemoveContainer" containerID="dbf32ba40689e755ac7a70b3822931f131352e80384eb04c5f62191003d74273" Nov 01 06:25:34.751282 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:34.751266 5040 scope.go:122] "RemoveContainer" containerID="dbf32ba40689e755ac7a70b3822931f131352e80384eb04c5f62191003d74273" Nov 01 06:25:34.751592 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:25:34.751572 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbf32ba40689e755ac7a70b3822931f131352e80384eb04c5f62191003d74273\": container with ID starting with dbf32ba40689e755ac7a70b3822931f131352e80384eb04c5f62191003d74273 not found: ID does not exist" containerID="dbf32ba40689e755ac7a70b3822931f131352e80384eb04c5f62191003d74273" Nov 01 06:25:34.751649 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:34.751601 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbf32ba40689e755ac7a70b3822931f131352e80384eb04c5f62191003d74273"} err="failed to get container status \"dbf32ba40689e755ac7a70b3822931f131352e80384eb04c5f62191003d74273\": rpc error: code = NotFound desc = could not find container \"dbf32ba40689e755ac7a70b3822931f131352e80384eb04c5f62191003d74273\": container with ID starting with dbf32ba40689e755ac7a70b3822931f131352e80384eb04c5f62191003d74273 not found: ID does not exist" Nov 01 06:25:34.777800 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:34.777769 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzkd7\" (UniqueName: \"kubernetes.io/projected/6e0b1e48-0511-4ef7-8f73-1106983b5b33-kube-api-access-lzkd7\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:34.778009 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:34.777983 5040 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6e0b1e48-0511-4ef7-8f73-1106983b5b33-tls\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:34.886991 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:34.886912 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-cd78f1/e2e-usage-sim-54291c-7c768bf864-pjszd"] Nov 01 06:25:34.892887 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:34.892863 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-cd78f1/e2e-usage-sim-54291c-7c768bf864-pjszd"] Nov 01 06:25:34.902117 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:34.902094 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e40-6t258" Nov 01 06:25:34.980270 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:34.980237 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fda7e954-48ee-4e6c-b04a-71df9940cb5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fda7e954-48ee-4e6c-b04a-71df9940cb5a-maas-api-openshift-ca\") pod \"fda7e954-48ee-4e6c-b04a-71df9940cb5a\" (UID: \"fda7e954-48ee-4e6c-b04a-71df9940cb5a\") " Nov 01 06:25:34.980428 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:34.980307 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fda7e954-48ee-4e6c-b04a-71df9940cb5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fda7e954-48ee-4e6c-b04a-71df9940cb5a-maas-api-xks-ca\") pod \"fda7e954-48ee-4e6c-b04a-71df9940cb5a\" (UID: \"fda7e954-48ee-4e6c-b04a-71df9940cb5a\") " Nov 01 06:25:34.980428 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:34.980347 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fda7e954-48ee-4e6c-b04a-71df9940cb5a-kube-api-access-pw6bd\" (UniqueName: \"kubernetes.io/projected/fda7e954-48ee-4e6c-b04a-71df9940cb5a-kube-api-access-pw6bd\") pod \"fda7e954-48ee-4e6c-b04a-71df9940cb5a\" (UID: \"fda7e954-48ee-4e6c-b04a-71df9940cb5a\") " Nov 01 06:25:34.980699 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:34.980673 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fda7e954-48ee-4e6c-b04a-71df9940cb5a-maas-api-openshift-ca" pod "fda7e954-48ee-4e6c-b04a-71df9940cb5a" (UID: "fda7e954-48ee-4e6c-b04a-71df9940cb5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:25:34.982382 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:34.982359 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fda7e954-48ee-4e6c-b04a-71df9940cb5a-maas-api-xks-ca" pod "fda7e954-48ee-4e6c-b04a-71df9940cb5a" (UID: "fda7e954-48ee-4e6c-b04a-71df9940cb5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:25:34.982441 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:34.982423 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fda7e954-48ee-4e6c-b04a-71df9940cb5a-kube-api-access-pw6bd" pod "fda7e954-48ee-4e6c-b04a-71df9940cb5a" (UID: "fda7e954-48ee-4e6c-b04a-71df9940cb5a"). InnerVolumeSpecName "kube-api-access-pw6bd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:25:35.081443 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:35.081405 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw6bd\" (UniqueName: \"kubernetes.io/projected/fda7e954-48ee-4e6c-b04a-71df9940cb5a-kube-api-access-pw6bd\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:35.081443 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:35.081437 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fda7e954-48ee-4e6c-b04a-71df9940cb5a-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:35.081443 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:35.081448 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fda7e954-48ee-4e6c-b04a-71df9940cb5a-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:35.514830 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:35.514795 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e0b1e48-0511-4ef7-8f73-1106983b5b33" path="/var/lib/kubelet/pods/6e0b1e48-0511-4ef7-8f73-1106983b5b33/volumes" Nov 01 06:25:35.748251 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:35.748214 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e40-6t258" Nov 01 06:25:35.748693 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:35.748211 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e40-6t258" event={"ID":"fda7e954-48ee-4e6c-b04a-71df9940cb5a","Type":"ContainerDied","Data":"a1e2873033cb187eb73f77eba95bf3ba4c4b9a3257f4f33c7747cc7a306267cf"} Nov 01 06:25:35.748693 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:35.748366 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1e2873033cb187eb73f77eba95bf3ba4c4b9a3257f4f33c7747cc7a306267cf" Nov 01 06:25:39.973237 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:25:39.973195 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:25:40.670265 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:40.670223 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-618a5fc8-jb24m"] Nov 01 06:25:40.670750 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:40.670735 5040 cpu_manager.go:414] "containerMap: removing container" podUID="fda7e954-48ee-4e6c-b04a-71df9940cb5a" containerName="revoke-keys" Nov 01 06:25:40.670750 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:40.670750 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fda7e954-48ee-4e6c-b04a-71df9940cb5a" containerName="revoke-keys" Nov 01 06:25:40.670866 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:40.670779 5040 cpu_manager.go:414] "containerMap: removing container" podUID="6e0b1e48-0511-4ef7-8f73-1106983b5b33" containerName="sim" Nov 01 06:25:40.670866 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:40.670784 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e0b1e48-0511-4ef7-8f73-1106983b5b33" containerName="sim" Nov 01 06:25:40.670866 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:40.670857 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e0b1e48-0511-4ef7-8f73-1106983b5b33" containerName="sim" Nov 01 06:25:40.670969 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:40.670869 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="fda7e954-48ee-4e6c-b04a-71df9940cb5a" containerName="revoke-keys" Nov 01 06:25:40.675467 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:40.675448 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-618a5fc8-jb24m" Nov 01 06:25:40.731028 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:40.730987 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-618a5fc8-jb24m"] Nov 01 06:25:40.738309 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:40.738280 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:25:40.738309 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:40.738276 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:25:40.738540 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:40.738305 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:25:40.756804 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:40.756769 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbdd1108-90e1-4806-a56a-b1e310fdc62d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-618a5fc8-jb24m\" (UID: \"dbdd1108-90e1-4806-a56a-b1e310fdc62d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-618a5fc8-jb24m" Nov 01 06:25:40.756977 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:40.756823 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbdd1108-90e1-4806-a56a-b1e310fdc62d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-618a5fc8-jb24m\" (UID: \"dbdd1108-90e1-4806-a56a-b1e310fdc62d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-618a5fc8-jb24m" Nov 01 06:25:40.757025 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:40.757012 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k77br\" (UniqueName: \"kubernetes.io/projected/dbdd1108-90e1-4806-a56a-b1e310fdc62d-kube-api-access-k77br\") pod \"maas-api-revoke-sub-618a5fc8-jb24m\" (UID: \"dbdd1108-90e1-4806-a56a-b1e310fdc62d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-618a5fc8-jb24m" Nov 01 06:25:40.777559 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:40.777530 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:25:40.858053 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:40.858021 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k77br\" (UniqueName: \"kubernetes.io/projected/dbdd1108-90e1-4806-a56a-b1e310fdc62d-kube-api-access-k77br\") pod \"maas-api-revoke-sub-618a5fc8-jb24m\" (UID: \"dbdd1108-90e1-4806-a56a-b1e310fdc62d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-618a5fc8-jb24m" Nov 01 06:25:40.858256 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:40.858089 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbdd1108-90e1-4806-a56a-b1e310fdc62d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-618a5fc8-jb24m\" (UID: \"dbdd1108-90e1-4806-a56a-b1e310fdc62d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-618a5fc8-jb24m" Nov 01 06:25:40.858256 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:40.858120 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbdd1108-90e1-4806-a56a-b1e310fdc62d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-618a5fc8-jb24m\" (UID: \"dbdd1108-90e1-4806-a56a-b1e310fdc62d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-618a5fc8-jb24m" Nov 01 06:25:40.858804 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:40.858783 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbdd1108-90e1-4806-a56a-b1e310fdc62d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-618a5fc8-jb24m\" (UID: \"dbdd1108-90e1-4806-a56a-b1e310fdc62d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-618a5fc8-jb24m" Nov 01 06:25:40.860510 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:40.860466 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbdd1108-90e1-4806-a56a-b1e310fdc62d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-618a5fc8-jb24m\" (UID: \"dbdd1108-90e1-4806-a56a-b1e310fdc62d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-618a5fc8-jb24m" Nov 01 06:25:40.919550 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:40.919516 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k77br\" (UniqueName: \"kubernetes.io/projected/dbdd1108-90e1-4806-a56a-b1e310fdc62d-kube-api-access-k77br\") pod \"maas-api-revoke-sub-618a5fc8-jb24m\" (UID: \"dbdd1108-90e1-4806-a56a-b1e310fdc62d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-618a5fc8-jb24m" Nov 01 06:25:40.989442 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:40.989359 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-618a5fc8-jb24m" Nov 01 06:25:41.372384 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:41.372352 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-618a5fc8-jb24m"] Nov 01 06:25:41.372966 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:25:41.372942 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbdd1108_90e1_4806_a56a_b1e310fdc62d.slice/crio-0a7c68a471d19d6d26517ec43e5597d915d778e1047718424057cde62d14d04d WatchSource:0}: Error finding container 0a7c68a471d19d6d26517ec43e5597d915d778e1047718424057cde62d14d04d: Status 404 returned error can't find the container with id 0a7c68a471d19d6d26517ec43e5597d915d778e1047718424057cde62d14d04d Nov 01 06:25:41.790535 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:41.790502 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-618a5fc8-jb24m" event={"ID":"dbdd1108-90e1-4806-a56a-b1e310fdc62d","Type":"ContainerStarted","Data":"0a7c68a471d19d6d26517ec43e5597d915d778e1047718424057cde62d14d04d"} Nov 01 06:25:42.798227 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:42.798187 5040 generic.go:356] "Generic (PLEG): container finished" podID="dbdd1108-90e1-4806-a56a-b1e310fdc62d" containerID="7a35bf2e13324befe3d0e1691cb73dcc27ee2bb3728c4da31bcd6c584c427784" exitCode=0 Nov 01 06:25:42.798715 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:42.798280 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-618a5fc8-jb24m" event={"ID":"dbdd1108-90e1-4806-a56a-b1e310fdc62d","Type":"ContainerDied","Data":"7a35bf2e13324befe3d0e1691cb73dcc27ee2bb3728c4da31bcd6c584c427784"} Nov 01 06:25:42.967597 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:42.967565 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt"] Nov 01 06:25:42.974844 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:42.974823 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt" Nov 01 06:25:43.012669 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:43.012634 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt"] Nov 01 06:25:43.027864 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:43.027827 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"payload-processing-external-model-plugins-e2e-extmod-b-3986bf\"" Nov 01 06:25:43.028064 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:43.027894 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"openshift-service-ca.crt\"" Nov 01 06:25:43.028064 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:43.027907 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"routing-overlay\"" Nov 01 06:25:43.028064 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:43.027901 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"agc-cred-28460a5a0934\"" Nov 01 06:25:43.028064 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:43.027900 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"kube-root-ca.crt\"" Nov 01 06:25:43.028064 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:43.027968 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"payload-processing-external-model-e2e--cf3b8510-dockercfg-ftjt9\"" Nov 01 06:25:43.082170 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:43.082091 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6061826c-3f8a-429d-b7a8-e12777095ea7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt\" (UID: \"6061826c-3f8a-429d-b7a8-e12777095ea7\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt" Nov 01 06:25:43.082170 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:43.082133 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6061826c-3f8a-429d-b7a8-e12777095ea7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt\" (UID: \"6061826c-3f8a-429d-b7a8-e12777095ea7\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt" Nov 01 06:25:43.082170 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:43.082162 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbbgh\" (UniqueName: \"kubernetes.io/projected/6061826c-3f8a-429d-b7a8-e12777095ea7-kube-api-access-dbbgh\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt\" (UID: \"6061826c-3f8a-429d-b7a8-e12777095ea7\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt" Nov 01 06:25:43.082384 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:43.082324 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6061826c-3f8a-429d-b7a8-e12777095ea7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt\" (UID: \"6061826c-3f8a-429d-b7a8-e12777095ea7\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt" Nov 01 06:25:43.183584 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:43.183544 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6061826c-3f8a-429d-b7a8-e12777095ea7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt\" (UID: \"6061826c-3f8a-429d-b7a8-e12777095ea7\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt" Nov 01 06:25:43.183584 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:43.183588 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6061826c-3f8a-429d-b7a8-e12777095ea7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt\" (UID: \"6061826c-3f8a-429d-b7a8-e12777095ea7\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt" Nov 01 06:25:43.183818 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:43.183705 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbbgh\" (UniqueName: \"kubernetes.io/projected/6061826c-3f8a-429d-b7a8-e12777095ea7-kube-api-access-dbbgh\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt\" (UID: \"6061826c-3f8a-429d-b7a8-e12777095ea7\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt" Nov 01 06:25:43.183912 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:43.183891 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6061826c-3f8a-429d-b7a8-e12777095ea7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt\" (UID: \"6061826c-3f8a-429d-b7a8-e12777095ea7\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt" Nov 01 06:25:43.184205 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:43.184188 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6061826c-3f8a-429d-b7a8-e12777095ea7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt\" (UID: \"6061826c-3f8a-429d-b7a8-e12777095ea7\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt" Nov 01 06:25:43.184471 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:43.184453 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6061826c-3f8a-429d-b7a8-e12777095ea7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt\" (UID: \"6061826c-3f8a-429d-b7a8-e12777095ea7\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt" Nov 01 06:25:43.186380 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:43.186362 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6061826c-3f8a-429d-b7a8-e12777095ea7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt\" (UID: \"6061826c-3f8a-429d-b7a8-e12777095ea7\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt" Nov 01 06:25:43.245468 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:43.245434 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbbgh\" (UniqueName: \"kubernetes.io/projected/6061826c-3f8a-429d-b7a8-e12777095ea7-kube-api-access-dbbgh\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt\" (UID: \"6061826c-3f8a-429d-b7a8-e12777095ea7\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt" Nov 01 06:25:43.293545 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:43.293505 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt" Nov 01 06:25:43.677070 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:43.677037 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt"] Nov 01 06:25:43.677576 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:25:43.677546 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6061826c_3f8a_429d_b7a8_e12777095ea7.slice/crio-e5ee7615ad0cbd39fcee0b487e97a5f6ee4a395316d9c61f000ef9079dcce7ba WatchSource:0}: Error finding container e5ee7615ad0cbd39fcee0b487e97a5f6ee4a395316d9c61f000ef9079dcce7ba: Status 404 returned error can't find the container with id e5ee7615ad0cbd39fcee0b487e97a5f6ee4a395316d9c61f000ef9079dcce7ba Nov 01 06:25:43.807411 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:43.807373 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt" event={"ID":"6061826c-3f8a-429d-b7a8-e12777095ea7","Type":"ContainerStarted","Data":"28fda5550fa67d9b134ba102e0f426f12c9cc87fa6f524f923107a553c35838b"} Nov 01 06:25:43.807857 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:43.807433 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt" event={"ID":"6061826c-3f8a-429d-b7a8-e12777095ea7","Type":"ContainerStarted","Data":"e5ee7615ad0cbd39fcee0b487e97a5f6ee4a395316d9c61f000ef9079dcce7ba"} Nov 01 06:25:43.807857 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:43.807529 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt" Nov 01 06:25:43.927526 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:43.927420 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt" podStartSLOduration=1.927402937 podStartE2EDuration="1.927402937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:25:43.924768627 +0000 UTC m=+2422.579885806" watchObservedRunningTime="2026-11-01 06:25:43.927402937 +0000 UTC m=+2422.582520095" Nov 01 06:25:43.938961 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:43.938935 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-618a5fc8-jb24m" Nov 01 06:25:43.995076 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:43.995043 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbdd1108-90e1-4806-a56a-b1e310fdc62d-kube-api-access-k77br\" (UniqueName: \"kubernetes.io/projected/dbdd1108-90e1-4806-a56a-b1e310fdc62d-kube-api-access-k77br\") pod \"dbdd1108-90e1-4806-a56a-b1e310fdc62d\" (UID: \"dbdd1108-90e1-4806-a56a-b1e310fdc62d\") " Nov 01 06:25:43.995262 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:43.995186 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbdd1108-90e1-4806-a56a-b1e310fdc62d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbdd1108-90e1-4806-a56a-b1e310fdc62d-maas-api-openshift-ca\") pod \"dbdd1108-90e1-4806-a56a-b1e310fdc62d\" (UID: \"dbdd1108-90e1-4806-a56a-b1e310fdc62d\") " Nov 01 06:25:43.995262 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:43.995243 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbdd1108-90e1-4806-a56a-b1e310fdc62d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbdd1108-90e1-4806-a56a-b1e310fdc62d-maas-api-xks-ca\") pod \"dbdd1108-90e1-4806-a56a-b1e310fdc62d\" (UID: \"dbdd1108-90e1-4806-a56a-b1e310fdc62d\") " Nov 01 06:25:43.995595 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:43.995572 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbdd1108-90e1-4806-a56a-b1e310fdc62d-maas-api-openshift-ca" pod "dbdd1108-90e1-4806-a56a-b1e310fdc62d" (UID: "dbdd1108-90e1-4806-a56a-b1e310fdc62d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:25:43.997234 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:43.997207 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbdd1108-90e1-4806-a56a-b1e310fdc62d-kube-api-access-k77br" pod "dbdd1108-90e1-4806-a56a-b1e310fdc62d" (UID: "dbdd1108-90e1-4806-a56a-b1e310fdc62d"). InnerVolumeSpecName "kube-api-access-k77br". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:25:43.997366 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:43.997346 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbdd1108-90e1-4806-a56a-b1e310fdc62d-maas-api-xks-ca" pod "dbdd1108-90e1-4806-a56a-b1e310fdc62d" (UID: "dbdd1108-90e1-4806-a56a-b1e310fdc62d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:25:44.096825 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:44.096790 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbdd1108-90e1-4806-a56a-b1e310fdc62d-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:44.096825 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:44.096820 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbdd1108-90e1-4806-a56a-b1e310fdc62d-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:44.096825 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:44.096830 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k77br\" (UniqueName: \"kubernetes.io/projected/dbdd1108-90e1-4806-a56a-b1e310fdc62d-kube-api-access-k77br\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:44.815821 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:44.815786 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-618a5fc8-jb24m" Nov 01 06:25:44.816249 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:44.815849 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-618a5fc8-jb24m" event={"ID":"dbdd1108-90e1-4806-a56a-b1e310fdc62d","Type":"ContainerDied","Data":"0a7c68a471d19d6d26517ec43e5597d915d778e1047718424057cde62d14d04d"} Nov 01 06:25:44.816249 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:44.815912 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a7c68a471d19d6d26517ec43e5597d915d778e1047718424057cde62d14d04d" Nov 01 06:25:47.789123 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:47.789083 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09c6fd53-gtn2n"] Nov 01 06:25:47.789609 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:47.789595 5040 cpu_manager.go:414] "containerMap: removing container" podUID="dbdd1108-90e1-4806-a56a-b1e310fdc62d" containerName="revoke-keys" Nov 01 06:25:47.789669 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:47.789609 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbdd1108-90e1-4806-a56a-b1e310fdc62d" containerName="revoke-keys" Nov 01 06:25:47.789710 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:47.789693 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbdd1108-90e1-4806-a56a-b1e310fdc62d" containerName="revoke-keys" Nov 01 06:25:47.793016 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:47.792999 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c6fd53-gtn2n" Nov 01 06:25:47.835214 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:47.835180 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe3c4dcf-3d99-4132-ba43-d177632a6a2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09c6fd53-gtn2n\" (UID: \"fe3c4dcf-3d99-4132-ba43-d177632a6a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c6fd53-gtn2n" Nov 01 06:25:47.835377 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:47.835243 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r52ff\" (UniqueName: \"kubernetes.io/projected/fe3c4dcf-3d99-4132-ba43-d177632a6a2c-kube-api-access-r52ff\") pod \"maas-api-revoke-sub-09c6fd53-gtn2n\" (UID: \"fe3c4dcf-3d99-4132-ba43-d177632a6a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c6fd53-gtn2n" Nov 01 06:25:47.835377 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:47.835344 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe3c4dcf-3d99-4132-ba43-d177632a6a2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09c6fd53-gtn2n\" (UID: \"fe3c4dcf-3d99-4132-ba43-d177632a6a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c6fd53-gtn2n" Nov 01 06:25:47.850828 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:47.850799 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09c6fd53-gtn2n"] Nov 01 06:25:47.897085 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:47.897044 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:25:47.897085 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:47.897075 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:25:47.897306 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:47.897140 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:25:47.897306 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:47.897206 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:25:47.936384 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:47.936354 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r52ff\" (UniqueName: \"kubernetes.io/projected/fe3c4dcf-3d99-4132-ba43-d177632a6a2c-kube-api-access-r52ff\") pod \"maas-api-revoke-sub-09c6fd53-gtn2n\" (UID: \"fe3c4dcf-3d99-4132-ba43-d177632a6a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c6fd53-gtn2n" Nov 01 06:25:47.936598 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:47.936405 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe3c4dcf-3d99-4132-ba43-d177632a6a2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09c6fd53-gtn2n\" (UID: \"fe3c4dcf-3d99-4132-ba43-d177632a6a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c6fd53-gtn2n" Nov 01 06:25:47.936598 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:47.936524 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe3c4dcf-3d99-4132-ba43-d177632a6a2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09c6fd53-gtn2n\" (UID: \"fe3c4dcf-3d99-4132-ba43-d177632a6a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c6fd53-gtn2n" Nov 01 06:25:47.937130 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:47.937109 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe3c4dcf-3d99-4132-ba43-d177632a6a2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09c6fd53-gtn2n\" (UID: \"fe3c4dcf-3d99-4132-ba43-d177632a6a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c6fd53-gtn2n" Nov 01 06:25:47.938867 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:47.938846 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe3c4dcf-3d99-4132-ba43-d177632a6a2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09c6fd53-gtn2n\" (UID: \"fe3c4dcf-3d99-4132-ba43-d177632a6a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c6fd53-gtn2n" Nov 01 06:25:47.998433 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:47.998392 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r52ff\" (UniqueName: \"kubernetes.io/projected/fe3c4dcf-3d99-4132-ba43-d177632a6a2c-kube-api-access-r52ff\") pod \"maas-api-revoke-sub-09c6fd53-gtn2n\" (UID: \"fe3c4dcf-3d99-4132-ba43-d177632a6a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c6fd53-gtn2n" Nov 01 06:25:48.106613 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:48.106520 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c6fd53-gtn2n" Nov 01 06:25:48.483837 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:48.483810 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09c6fd53-gtn2n"] Nov 01 06:25:48.484352 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:25:48.484327 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe3c4dcf_3d99_4132_ba43_d177632a6a2c.slice/crio-56d6927108b5ca6c32c1c979af4f167c009d9e1dee68ec44939f15786781d87f WatchSource:0}: Error finding container 56d6927108b5ca6c32c1c979af4f167c009d9e1dee68ec44939f15786781d87f: Status 404 returned error can't find the container with id 56d6927108b5ca6c32c1c979af4f167c009d9e1dee68ec44939f15786781d87f Nov 01 06:25:48.876701 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:48.876660 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c6fd53-gtn2n" event={"ID":"fe3c4dcf-3d99-4132-ba43-d177632a6a2c","Type":"ContainerStarted","Data":"573eb279e5a63355b8a3ea7c7e78b68429fb6b9b4a46f32031fb05c948e54db5"} Nov 01 06:25:48.877084 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:48.876719 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c6fd53-gtn2n" event={"ID":"fe3c4dcf-3d99-4132-ba43-d177632a6a2c","Type":"ContainerStarted","Data":"56d6927108b5ca6c32c1c979af4f167c009d9e1dee68ec44939f15786781d87f"} Nov 01 06:25:48.994499 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:48.994424 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c6fd53-gtn2n" podStartSLOduration=1.994411197 podStartE2EDuration="1.994411197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:25:48.993761628 +0000 UTC m=+2427.648878796" watchObservedRunningTime="2026-11-01 06:25:48.994411197 +0000 UTC m=+2427.649528364" Nov 01 06:25:49.885153 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:49.885117 5040 generic.go:356] "Generic (PLEG): container finished" podID="fe3c4dcf-3d99-4132-ba43-d177632a6a2c" containerID="573eb279e5a63355b8a3ea7c7e78b68429fb6b9b4a46f32031fb05c948e54db5" exitCode=0 Nov 01 06:25:49.885614 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:49.885162 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c6fd53-gtn2n" event={"ID":"fe3c4dcf-3d99-4132-ba43-d177632a6a2c","Type":"ContainerDied","Data":"573eb279e5a63355b8a3ea7c7e78b68429fb6b9b4a46f32031fb05c948e54db5"} Nov 01 06:25:51.043563 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:51.043537 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c6fd53-gtn2n" Nov 01 06:25:51.171967 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:51.171889 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe3c4dcf-3d99-4132-ba43-d177632a6a2c-kube-api-access-r52ff\" (UniqueName: \"kubernetes.io/projected/fe3c4dcf-3d99-4132-ba43-d177632a6a2c-kube-api-access-r52ff\") pod \"fe3c4dcf-3d99-4132-ba43-d177632a6a2c\" (UID: \"fe3c4dcf-3d99-4132-ba43-d177632a6a2c\") " Nov 01 06:25:51.172151 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:51.172012 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe3c4dcf-3d99-4132-ba43-d177632a6a2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe3c4dcf-3d99-4132-ba43-d177632a6a2c-maas-api-openshift-ca\") pod \"fe3c4dcf-3d99-4132-ba43-d177632a6a2c\" (UID: \"fe3c4dcf-3d99-4132-ba43-d177632a6a2c\") " Nov 01 06:25:51.172151 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:51.172044 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe3c4dcf-3d99-4132-ba43-d177632a6a2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe3c4dcf-3d99-4132-ba43-d177632a6a2c-maas-api-xks-ca\") pod \"fe3c4dcf-3d99-4132-ba43-d177632a6a2c\" (UID: \"fe3c4dcf-3d99-4132-ba43-d177632a6a2c\") " Nov 01 06:25:51.172487 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:51.172445 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe3c4dcf-3d99-4132-ba43-d177632a6a2c-maas-api-openshift-ca" pod "fe3c4dcf-3d99-4132-ba43-d177632a6a2c" (UID: "fe3c4dcf-3d99-4132-ba43-d177632a6a2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:25:51.174186 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:51.174157 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe3c4dcf-3d99-4132-ba43-d177632a6a2c-kube-api-access-r52ff" pod "fe3c4dcf-3d99-4132-ba43-d177632a6a2c" (UID: "fe3c4dcf-3d99-4132-ba43-d177632a6a2c"). InnerVolumeSpecName "kube-api-access-r52ff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:25:51.174686 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:51.174665 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe3c4dcf-3d99-4132-ba43-d177632a6a2c-maas-api-xks-ca" pod "fe3c4dcf-3d99-4132-ba43-d177632a6a2c" (UID: "fe3c4dcf-3d99-4132-ba43-d177632a6a2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:25:51.273924 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:51.273889 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r52ff\" (UniqueName: \"kubernetes.io/projected/fe3c4dcf-3d99-4132-ba43-d177632a6a2c-kube-api-access-r52ff\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:51.273924 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:51.273918 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe3c4dcf-3d99-4132-ba43-d177632a6a2c-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:51.273924 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:51.273929 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe3c4dcf-3d99-4132-ba43-d177632a6a2c-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:51.899512 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:51.899412 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c6fd53-gtn2n" Nov 01 06:25:51.899512 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:51.899456 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c6fd53-gtn2n" event={"ID":"fe3c4dcf-3d99-4132-ba43-d177632a6a2c","Type":"ContainerDied","Data":"56d6927108b5ca6c32c1c979af4f167c009d9e1dee68ec44939f15786781d87f"} Nov 01 06:25:51.899746 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:51.899527 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56d6927108b5ca6c32c1c979af4f167c009d9e1dee68ec44939f15786781d87f" Nov 01 06:25:54.444986 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:54.444953 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt"] Nov 01 06:25:54.447615 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:54.447579 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt" podUID="6061826c-3f8a-429d-b7a8-e12777095ea7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:25:59.309512 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.309460 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a19f8bb-slrvd"] Nov 01 06:25:59.309993 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.309980 5040 cpu_manager.go:414] "containerMap: removing container" podUID="fe3c4dcf-3d99-4132-ba43-d177632a6a2c" containerName="revoke-keys" Nov 01 06:25:59.310039 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.309995 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe3c4dcf-3d99-4132-ba43-d177632a6a2c" containerName="revoke-keys" Nov 01 06:25:59.310104 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.310095 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe3c4dcf-3d99-4132-ba43-d177632a6a2c" containerName="revoke-keys" Nov 01 06:25:59.313756 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.313735 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a19f8bb-slrvd" Nov 01 06:25:59.370871 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.370836 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a19f8bb-slrvd"] Nov 01 06:25:59.417001 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.416964 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:25:59.417001 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.416988 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:25:59.417001 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.416975 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:25:59.417256 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.417031 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:25:59.445989 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.445931 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt" podUID="6061826c-3f8a-429d-b7a8-e12777095ea7" containerName="payload-processing" containerID="cri-o://28fda5550fa67d9b134ba102e0f426f12c9cc87fa6f524f923107a553c35838b" gracePeriod=25 Nov 01 06:25:59.486948 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.486916 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92f94f44-2b25-4809-a38d-e3eabab49544-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a19f8bb-slrvd\" (UID: \"92f94f44-2b25-4809-a38d-e3eabab49544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a19f8bb-slrvd" Nov 01 06:25:59.486948 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.486960 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92f94f44-2b25-4809-a38d-e3eabab49544-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a19f8bb-slrvd\" (UID: \"92f94f44-2b25-4809-a38d-e3eabab49544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a19f8bb-slrvd" Nov 01 06:25:59.487163 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.487095 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t6jz\" (UniqueName: \"kubernetes.io/projected/92f94f44-2b25-4809-a38d-e3eabab49544-kube-api-access-5t6jz\") pod \"maas-api-revoke-sub-6a19f8bb-slrvd\" (UID: \"92f94f44-2b25-4809-a38d-e3eabab49544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a19f8bb-slrvd" Nov 01 06:25:59.587824 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.587784 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92f94f44-2b25-4809-a38d-e3eabab49544-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a19f8bb-slrvd\" (UID: \"92f94f44-2b25-4809-a38d-e3eabab49544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a19f8bb-slrvd" Nov 01 06:25:59.587994 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.587845 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92f94f44-2b25-4809-a38d-e3eabab49544-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a19f8bb-slrvd\" (UID: \"92f94f44-2b25-4809-a38d-e3eabab49544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a19f8bb-slrvd" Nov 01 06:25:59.587994 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.587972 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t6jz\" (UniqueName: \"kubernetes.io/projected/92f94f44-2b25-4809-a38d-e3eabab49544-kube-api-access-5t6jz\") pod \"maas-api-revoke-sub-6a19f8bb-slrvd\" (UID: \"92f94f44-2b25-4809-a38d-e3eabab49544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a19f8bb-slrvd" Nov 01 06:25:59.588683 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.588656 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92f94f44-2b25-4809-a38d-e3eabab49544-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a19f8bb-slrvd\" (UID: \"92f94f44-2b25-4809-a38d-e3eabab49544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a19f8bb-slrvd" Nov 01 06:25:59.590186 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.590167 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92f94f44-2b25-4809-a38d-e3eabab49544-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a19f8bb-slrvd\" (UID: \"92f94f44-2b25-4809-a38d-e3eabab49544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a19f8bb-slrvd" Nov 01 06:25:59.652020 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.651991 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t6jz\" (UniqueName: \"kubernetes.io/projected/92f94f44-2b25-4809-a38d-e3eabab49544-kube-api-access-5t6jz\") pod \"maas-api-revoke-sub-6a19f8bb-slrvd\" (UID: \"92f94f44-2b25-4809-a38d-e3eabab49544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a19f8bb-slrvd" Nov 01 06:25:59.742956 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.742926 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt" Nov 01 06:25:59.894384 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.894298 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6061826c-3f8a-429d-b7a8-e12777095ea7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6061826c-3f8a-429d-b7a8-e12777095ea7-plugins-config-volume\") pod \"6061826c-3f8a-429d-b7a8-e12777095ea7\" (UID: \"6061826c-3f8a-429d-b7a8-e12777095ea7\") " Nov 01 06:25:59.894384 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.894355 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6061826c-3f8a-429d-b7a8-e12777095ea7-kube-api-access-dbbgh\" (UniqueName: \"kubernetes.io/projected/6061826c-3f8a-429d-b7a8-e12777095ea7-kube-api-access-dbbgh\") pod \"6061826c-3f8a-429d-b7a8-e12777095ea7\" (UID: \"6061826c-3f8a-429d-b7a8-e12777095ea7\") " Nov 01 06:25:59.894672 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.894467 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6061826c-3f8a-429d-b7a8-e12777095ea7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6061826c-3f8a-429d-b7a8-e12777095ea7-provider-credentials\") pod \"6061826c-3f8a-429d-b7a8-e12777095ea7\" (UID: \"6061826c-3f8a-429d-b7a8-e12777095ea7\") " Nov 01 06:25:59.894672 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.894548 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6061826c-3f8a-429d-b7a8-e12777095ea7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6061826c-3f8a-429d-b7a8-e12777095ea7-routing-overlay-volume\") pod \"6061826c-3f8a-429d-b7a8-e12777095ea7\" (UID: \"6061826c-3f8a-429d-b7a8-e12777095ea7\") " Nov 01 06:25:59.894841 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.894808 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6061826c-3f8a-429d-b7a8-e12777095ea7-plugins-config-volume" pod "6061826c-3f8a-429d-b7a8-e12777095ea7" (UID: "6061826c-3f8a-429d-b7a8-e12777095ea7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:25:59.894978 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.894939 5040 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6061826c-3f8a-429d-b7a8-e12777095ea7-plugins-config-volume\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:59.894978 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.894946 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6061826c-3f8a-429d-b7a8-e12777095ea7-routing-overlay-volume" pod "6061826c-3f8a-429d-b7a8-e12777095ea7" (UID: "6061826c-3f8a-429d-b7a8-e12777095ea7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:25:59.897188 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.897161 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6061826c-3f8a-429d-b7a8-e12777095ea7-provider-credentials" pod "6061826c-3f8a-429d-b7a8-e12777095ea7" (UID: "6061826c-3f8a-429d-b7a8-e12777095ea7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:25:59.897287 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.897168 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6061826c-3f8a-429d-b7a8-e12777095ea7-kube-api-access-dbbgh" pod "6061826c-3f8a-429d-b7a8-e12777095ea7" (UID: "6061826c-3f8a-429d-b7a8-e12777095ea7"). InnerVolumeSpecName "kube-api-access-dbbgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:25:59.927494 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.927450 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a19f8bb-slrvd" Nov 01 06:25:59.955515 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.955460 5040 generic.go:356] "Generic (PLEG): container finished" podID="6061826c-3f8a-429d-b7a8-e12777095ea7" containerID="28fda5550fa67d9b134ba102e0f426f12c9cc87fa6f524f923107a553c35838b" exitCode=0 Nov 01 06:25:59.955681 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.955574 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt" Nov 01 06:25:59.955681 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.955596 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt" event={"ID":"6061826c-3f8a-429d-b7a8-e12777095ea7","Type":"ContainerDied","Data":"28fda5550fa67d9b134ba102e0f426f12c9cc87fa6f524f923107a553c35838b"} Nov 01 06:25:59.955681 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.955658 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt" event={"ID":"6061826c-3f8a-429d-b7a8-e12777095ea7","Type":"ContainerDied","Data":"e5ee7615ad0cbd39fcee0b487e97a5f6ee4a395316d9c61f000ef9079dcce7ba"} Nov 01 06:25:59.955942 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.955691 5040 scope.go:122] "RemoveContainer" containerID="28fda5550fa67d9b134ba102e0f426f12c9cc87fa6f524f923107a553c35838b" Nov 01 06:25:59.969070 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.969046 5040 scope.go:122] "RemoveContainer" containerID="28fda5550fa67d9b134ba102e0f426f12c9cc87fa6f524f923107a553c35838b" Nov 01 06:25:59.969394 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:25:59.969377 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28fda5550fa67d9b134ba102e0f426f12c9cc87fa6f524f923107a553c35838b\": container with ID starting with 28fda5550fa67d9b134ba102e0f426f12c9cc87fa6f524f923107a553c35838b not found: ID does not exist" containerID="28fda5550fa67d9b134ba102e0f426f12c9cc87fa6f524f923107a553c35838b" Nov 01 06:25:59.969436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.969402 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28fda5550fa67d9b134ba102e0f426f12c9cc87fa6f524f923107a553c35838b"} err="failed to get container status \"28fda5550fa67d9b134ba102e0f426f12c9cc87fa6f524f923107a553c35838b\": rpc error: code = NotFound desc = could not find container \"28fda5550fa67d9b134ba102e0f426f12c9cc87fa6f524f923107a553c35838b\": container with ID starting with 28fda5550fa67d9b134ba102e0f426f12c9cc87fa6f524f923107a553c35838b not found: ID does not exist" Nov 01 06:25:59.996135 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.996102 5040 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6061826c-3f8a-429d-b7a8-e12777095ea7-routing-overlay-volume\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:59.996135 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.996133 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbbgh\" (UniqueName: \"kubernetes.io/projected/6061826c-3f8a-429d-b7a8-e12777095ea7-kube-api-access-dbbgh\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:59.996357 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:25:59.996142 5040 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6061826c-3f8a-429d-b7a8-e12777095ea7-provider-credentials\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:00.123691 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:00.123590 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt"] Nov 01 06:26:00.127296 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:00.127270 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-cd7dkfvxt"] Nov 01 06:26:00.311488 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:00.311447 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a19f8bb-slrvd"] Nov 01 06:26:00.311890 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:26:00.311451 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92f94f44_2b25_4809_a38d_e3eabab49544.slice/crio-ca1e4c86bd666c01274277acee5cb90074fb29969b9344259a72c10ac5b6e729 WatchSource:0}: Error finding container ca1e4c86bd666c01274277acee5cb90074fb29969b9344259a72c10ac5b6e729: Status 404 returned error can't find the container with id ca1e4c86bd666c01274277acee5cb90074fb29969b9344259a72c10ac5b6e729 Nov 01 06:26:00.966128 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:00.966029 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a19f8bb-slrvd" event={"ID":"92f94f44-2b25-4809-a38d-e3eabab49544","Type":"ContainerStarted","Data":"9aa305c39370e675c2ba2589af43f496757d652419e799bd4dc51405761ccfae"} Nov 01 06:26:00.966128 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:00.966090 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a19f8bb-slrvd" event={"ID":"92f94f44-2b25-4809-a38d-e3eabab49544","Type":"ContainerStarted","Data":"ca1e4c86bd666c01274277acee5cb90074fb29969b9344259a72c10ac5b6e729"} Nov 01 06:26:01.086353 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:01.086287 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a19f8bb-slrvd" podStartSLOduration=2.086272838 podStartE2EDuration="2.086272838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:25:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:26:01.082724272 +0000 UTC m=+2439.737841440" watchObservedRunningTime="2026-11-01 06:26:01.086272838 +0000 UTC m=+2439.741390004" Nov 01 06:26:01.515315 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:01.515282 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6061826c-3f8a-429d-b7a8-e12777095ea7" path="/var/lib/kubelet/pods/6061826c-3f8a-429d-b7a8-e12777095ea7/volumes" Nov 01 06:26:01.973792 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:01.973706 5040 generic.go:356] "Generic (PLEG): container finished" podID="92f94f44-2b25-4809-a38d-e3eabab49544" containerID="9aa305c39370e675c2ba2589af43f496757d652419e799bd4dc51405761ccfae" exitCode=0 Nov 01 06:26:01.973792 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:01.973785 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a19f8bb-slrvd" event={"ID":"92f94f44-2b25-4809-a38d-e3eabab49544","Type":"ContainerDied","Data":"9aa305c39370e675c2ba2589af43f496757d652419e799bd4dc51405761ccfae"} Nov 01 06:26:03.131046 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:03.129806 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a19f8bb-slrvd" Nov 01 06:26:03.231177 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:03.231087 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92f94f44-2b25-4809-a38d-e3eabab49544-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92f94f44-2b25-4809-a38d-e3eabab49544-maas-api-xks-ca\") pod \"92f94f44-2b25-4809-a38d-e3eabab49544\" (UID: \"92f94f44-2b25-4809-a38d-e3eabab49544\") " Nov 01 06:26:03.231339 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:03.231270 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92f94f44-2b25-4809-a38d-e3eabab49544-kube-api-access-5t6jz\" (UniqueName: \"kubernetes.io/projected/92f94f44-2b25-4809-a38d-e3eabab49544-kube-api-access-5t6jz\") pod \"92f94f44-2b25-4809-a38d-e3eabab49544\" (UID: \"92f94f44-2b25-4809-a38d-e3eabab49544\") " Nov 01 06:26:03.231339 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:03.231307 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92f94f44-2b25-4809-a38d-e3eabab49544-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92f94f44-2b25-4809-a38d-e3eabab49544-maas-api-openshift-ca\") pod \"92f94f44-2b25-4809-a38d-e3eabab49544\" (UID: \"92f94f44-2b25-4809-a38d-e3eabab49544\") " Nov 01 06:26:03.231700 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:03.231671 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92f94f44-2b25-4809-a38d-e3eabab49544-maas-api-openshift-ca" pod "92f94f44-2b25-4809-a38d-e3eabab49544" (UID: "92f94f44-2b25-4809-a38d-e3eabab49544"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:26:03.233348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:03.233316 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92f94f44-2b25-4809-a38d-e3eabab49544-kube-api-access-5t6jz" pod "92f94f44-2b25-4809-a38d-e3eabab49544" (UID: "92f94f44-2b25-4809-a38d-e3eabab49544"). InnerVolumeSpecName "kube-api-access-5t6jz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:26:03.233448 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:03.233418 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92f94f44-2b25-4809-a38d-e3eabab49544-maas-api-xks-ca" pod "92f94f44-2b25-4809-a38d-e3eabab49544" (UID: "92f94f44-2b25-4809-a38d-e3eabab49544"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:26:03.332898 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:03.332743 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t6jz\" (UniqueName: \"kubernetes.io/projected/92f94f44-2b25-4809-a38d-e3eabab49544-kube-api-access-5t6jz\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:03.332898 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:03.332777 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92f94f44-2b25-4809-a38d-e3eabab49544-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:03.332898 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:03.332788 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92f94f44-2b25-4809-a38d-e3eabab49544-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:03.987755 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:03.987667 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a19f8bb-slrvd" Nov 01 06:26:03.987755 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:03.987703 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a19f8bb-slrvd" event={"ID":"92f94f44-2b25-4809-a38d-e3eabab49544","Type":"ContainerDied","Data":"ca1e4c86bd666c01274277acee5cb90074fb29969b9344259a72c10ac5b6e729"} Nov 01 06:26:03.987755 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:03.987752 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca1e4c86bd666c01274277acee5cb90074fb29969b9344259a72c10ac5b6e729" Nov 01 06:26:09.973392 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:26:09.973353 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:26:12.913856 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:12.913813 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac6348fe-zvftk"] Nov 01 06:26:12.914642 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:12.914622 5040 cpu_manager.go:414] "containerMap: removing container" podUID="92f94f44-2b25-4809-a38d-e3eabab49544" containerName="revoke-keys" Nov 01 06:26:12.914713 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:12.914646 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92f94f44-2b25-4809-a38d-e3eabab49544" containerName="revoke-keys" Nov 01 06:26:12.914713 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:12.914669 5040 cpu_manager.go:414] "containerMap: removing container" podUID="6061826c-3f8a-429d-b7a8-e12777095ea7" containerName="payload-processing" Nov 01 06:26:12.914713 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:12.914678 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6061826c-3f8a-429d-b7a8-e12777095ea7" containerName="payload-processing" Nov 01 06:26:12.914841 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:12.914828 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="92f94f44-2b25-4809-a38d-e3eabab49544" containerName="revoke-keys" Nov 01 06:26:12.914876 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:12.914847 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="6061826c-3f8a-429d-b7a8-e12777095ea7" containerName="payload-processing" Nov 01 06:26:12.919039 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:12.919016 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac6348fe-zvftk" Nov 01 06:26:12.972627 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:12.972596 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac6348fe-zvftk"] Nov 01 06:26:13.019305 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:13.019165 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:26:13.019528 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:13.019277 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:26:13.019528 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:13.019283 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:26:13.019528 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:13.019292 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:26:13.036337 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:13.036297 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln9qp\" (UniqueName: \"kubernetes.io/projected/25fdfaf7-a907-4d2e-9786-93a5db503d2b-kube-api-access-ln9qp\") pod \"maas-api-revoke-sub-ac6348fe-zvftk\" (UID: \"25fdfaf7-a907-4d2e-9786-93a5db503d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac6348fe-zvftk" Nov 01 06:26:13.036589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:13.036466 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25fdfaf7-a907-4d2e-9786-93a5db503d2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac6348fe-zvftk\" (UID: \"25fdfaf7-a907-4d2e-9786-93a5db503d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac6348fe-zvftk" Nov 01 06:26:13.036589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:13.036533 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25fdfaf7-a907-4d2e-9786-93a5db503d2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac6348fe-zvftk\" (UID: \"25fdfaf7-a907-4d2e-9786-93a5db503d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac6348fe-zvftk" Nov 01 06:26:13.137442 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:13.137395 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln9qp\" (UniqueName: \"kubernetes.io/projected/25fdfaf7-a907-4d2e-9786-93a5db503d2b-kube-api-access-ln9qp\") pod \"maas-api-revoke-sub-ac6348fe-zvftk\" (UID: \"25fdfaf7-a907-4d2e-9786-93a5db503d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac6348fe-zvftk" Nov 01 06:26:13.137676 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:13.137504 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25fdfaf7-a907-4d2e-9786-93a5db503d2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac6348fe-zvftk\" (UID: \"25fdfaf7-a907-4d2e-9786-93a5db503d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac6348fe-zvftk" Nov 01 06:26:13.137676 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:13.137532 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25fdfaf7-a907-4d2e-9786-93a5db503d2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac6348fe-zvftk\" (UID: \"25fdfaf7-a907-4d2e-9786-93a5db503d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac6348fe-zvftk" Nov 01 06:26:13.138444 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:13.138415 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25fdfaf7-a907-4d2e-9786-93a5db503d2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac6348fe-zvftk\" (UID: \"25fdfaf7-a907-4d2e-9786-93a5db503d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac6348fe-zvftk" Nov 01 06:26:13.140183 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:13.140160 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25fdfaf7-a907-4d2e-9786-93a5db503d2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac6348fe-zvftk\" (UID: \"25fdfaf7-a907-4d2e-9786-93a5db503d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac6348fe-zvftk" Nov 01 06:26:13.199430 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:13.199334 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln9qp\" (UniqueName: \"kubernetes.io/projected/25fdfaf7-a907-4d2e-9786-93a5db503d2b-kube-api-access-ln9qp\") pod \"maas-api-revoke-sub-ac6348fe-zvftk\" (UID: \"25fdfaf7-a907-4d2e-9786-93a5db503d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac6348fe-zvftk" Nov 01 06:26:13.237017 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:13.236968 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac6348fe-zvftk" Nov 01 06:26:13.621074 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:13.621041 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac6348fe-zvftk"] Nov 01 06:26:13.621222 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:26:13.621178 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25fdfaf7_a907_4d2e_9786_93a5db503d2b.slice/crio-0cf788417ee047a786b70343a2aad713202a5a7cc81d822f1384cd4c18578ba7 WatchSource:0}: Error finding container 0cf788417ee047a786b70343a2aad713202a5a7cc81d822f1384cd4c18578ba7: Status 404 returned error can't find the container with id 0cf788417ee047a786b70343a2aad713202a5a7cc81d822f1384cd4c18578ba7 Nov 01 06:26:14.064887 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:14.064850 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac6348fe-zvftk" event={"ID":"25fdfaf7-a907-4d2e-9786-93a5db503d2b","Type":"ContainerStarted","Data":"2332a69512af9dce998a62285a4765d60fabe5e67447e55cfbb4d9f965ce52a0"} Nov 01 06:26:14.065278 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:14.064907 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac6348fe-zvftk" event={"ID":"25fdfaf7-a907-4d2e-9786-93a5db503d2b","Type":"ContainerStarted","Data":"0cf788417ee047a786b70343a2aad713202a5a7cc81d822f1384cd4c18578ba7"} Nov 01 06:26:14.234922 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:14.234817 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac6348fe-zvftk" podStartSLOduration=2.234801417 podStartE2EDuration="2.234801417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:26:14.182655595 +0000 UTC m=+2452.837772762" watchObservedRunningTime="2026-11-01 06:26:14.234801417 +0000 UTC m=+2452.889918585" Nov 01 06:26:15.073459 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:15.073419 5040 generic.go:356] "Generic (PLEG): container finished" podID="25fdfaf7-a907-4d2e-9786-93a5db503d2b" containerID="2332a69512af9dce998a62285a4765d60fabe5e67447e55cfbb4d9f965ce52a0" exitCode=0 Nov 01 06:26:15.073870 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:15.073507 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac6348fe-zvftk" event={"ID":"25fdfaf7-a907-4d2e-9786-93a5db503d2b","Type":"ContainerDied","Data":"2332a69512af9dce998a62285a4765d60fabe5e67447e55cfbb4d9f965ce52a0"} Nov 01 06:26:16.227239 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:16.227215 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac6348fe-zvftk" Nov 01 06:26:16.293978 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:16.293944 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25fdfaf7-a907-4d2e-9786-93a5db503d2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25fdfaf7-a907-4d2e-9786-93a5db503d2b-maas-api-openshift-ca\") pod \"25fdfaf7-a907-4d2e-9786-93a5db503d2b\" (UID: \"25fdfaf7-a907-4d2e-9786-93a5db503d2b\") " Nov 01 06:26:16.294158 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:16.293993 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25fdfaf7-a907-4d2e-9786-93a5db503d2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25fdfaf7-a907-4d2e-9786-93a5db503d2b-maas-api-xks-ca\") pod \"25fdfaf7-a907-4d2e-9786-93a5db503d2b\" (UID: \"25fdfaf7-a907-4d2e-9786-93a5db503d2b\") " Nov 01 06:26:16.294158 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:16.294044 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25fdfaf7-a907-4d2e-9786-93a5db503d2b-kube-api-access-ln9qp\" (UniqueName: \"kubernetes.io/projected/25fdfaf7-a907-4d2e-9786-93a5db503d2b-kube-api-access-ln9qp\") pod \"25fdfaf7-a907-4d2e-9786-93a5db503d2b\" (UID: \"25fdfaf7-a907-4d2e-9786-93a5db503d2b\") " Nov 01 06:26:16.294309 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:16.294287 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25fdfaf7-a907-4d2e-9786-93a5db503d2b-maas-api-openshift-ca" pod "25fdfaf7-a907-4d2e-9786-93a5db503d2b" (UID: "25fdfaf7-a907-4d2e-9786-93a5db503d2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:26:16.296106 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:16.296081 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25fdfaf7-a907-4d2e-9786-93a5db503d2b-kube-api-access-ln9qp" pod "25fdfaf7-a907-4d2e-9786-93a5db503d2b" (UID: "25fdfaf7-a907-4d2e-9786-93a5db503d2b"). InnerVolumeSpecName "kube-api-access-ln9qp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:26:16.296200 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:16.296148 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25fdfaf7-a907-4d2e-9786-93a5db503d2b-maas-api-xks-ca" pod "25fdfaf7-a907-4d2e-9786-93a5db503d2b" (UID: "25fdfaf7-a907-4d2e-9786-93a5db503d2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:26:16.395018 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:16.394926 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25fdfaf7-a907-4d2e-9786-93a5db503d2b-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:16.395018 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:16.394960 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25fdfaf7-a907-4d2e-9786-93a5db503d2b-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:16.395018 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:16.394973 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln9qp\" (UniqueName: \"kubernetes.io/projected/25fdfaf7-a907-4d2e-9786-93a5db503d2b-kube-api-access-ln9qp\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:17.093016 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:17.092984 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac6348fe-zvftk" Nov 01 06:26:17.093202 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:17.093025 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac6348fe-zvftk" event={"ID":"25fdfaf7-a907-4d2e-9786-93a5db503d2b","Type":"ContainerDied","Data":"0cf788417ee047a786b70343a2aad713202a5a7cc81d822f1384cd4c18578ba7"} Nov 01 06:26:17.093202 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:17.093114 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cf788417ee047a786b70343a2aad713202a5a7cc81d822f1384cd4c18578ba7" Nov 01 06:26:17.741798 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:17.741759 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb"] Nov 01 06:26:17.742518 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:17.742496 5040 cpu_manager.go:414] "containerMap: removing container" podUID="25fdfaf7-a907-4d2e-9786-93a5db503d2b" containerName="revoke-keys" Nov 01 06:26:17.742518 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:17.742519 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25fdfaf7-a907-4d2e-9786-93a5db503d2b" containerName="revoke-keys" Nov 01 06:26:17.742677 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:17.742661 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="25fdfaf7-a907-4d2e-9786-93a5db503d2b" containerName="revoke-keys" Nov 01 06:26:17.746310 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:17.746288 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb" Nov 01 06:26:17.804901 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:17.804865 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb"] Nov 01 06:26:17.805563 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:17.805539 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5dc4dc82-c7b4-4043-b402-6809e9429104-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb\" (UID: \"5dc4dc82-c7b4-4043-b402-6809e9429104\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb" Nov 01 06:26:17.805664 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:17.805593 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9sgq\" (UniqueName: \"kubernetes.io/projected/5dc4dc82-c7b4-4043-b402-6809e9429104-kube-api-access-z9sgq\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb\" (UID: \"5dc4dc82-c7b4-4043-b402-6809e9429104\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb" Nov 01 06:26:17.805723 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:17.805705 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc4dc82-c7b4-4043-b402-6809e9429104-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb\" (UID: \"5dc4dc82-c7b4-4043-b402-6809e9429104\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb" Nov 01 06:26:17.805797 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:17.805783 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5dc4dc82-c7b4-4043-b402-6809e9429104-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb\" (UID: \"5dc4dc82-c7b4-4043-b402-6809e9429104\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb" Nov 01 06:26:17.850934 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:17.850899 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"agc-cred-d6d3484085b2\"" Nov 01 06:26:17.850934 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:17.850925 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"routing-overlay\"" Nov 01 06:26:17.851166 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:17.850943 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"openshift-service-ca.crt\"" Nov 01 06:26:17.851166 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:17.850945 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"payload-processing-external-model-e2e--cb1d2723-dockercfg-smk8q\"" Nov 01 06:26:17.851166 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:17.850909 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"payload-processing-external-model-plugins-e2e-extmod-a-8f45a2\"" Nov 01 06:26:17.851166 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:17.850988 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"kube-root-ca.crt\"" Nov 01 06:26:17.906291 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:17.906250 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc4dc82-c7b4-4043-b402-6809e9429104-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb\" (UID: \"5dc4dc82-c7b4-4043-b402-6809e9429104\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb" Nov 01 06:26:17.906525 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:17.906347 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5dc4dc82-c7b4-4043-b402-6809e9429104-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb\" (UID: \"5dc4dc82-c7b4-4043-b402-6809e9429104\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb" Nov 01 06:26:17.906525 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:17.906419 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5dc4dc82-c7b4-4043-b402-6809e9429104-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb\" (UID: \"5dc4dc82-c7b4-4043-b402-6809e9429104\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb" Nov 01 06:26:17.906525 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:17.906466 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9sgq\" (UniqueName: \"kubernetes.io/projected/5dc4dc82-c7b4-4043-b402-6809e9429104-kube-api-access-z9sgq\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb\" (UID: \"5dc4dc82-c7b4-4043-b402-6809e9429104\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb" Nov 01 06:26:17.907125 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:17.907100 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5dc4dc82-c7b4-4043-b402-6809e9429104-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb\" (UID: \"5dc4dc82-c7b4-4043-b402-6809e9429104\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb" Nov 01 06:26:17.907201 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:17.907109 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc4dc82-c7b4-4043-b402-6809e9429104-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb\" (UID: \"5dc4dc82-c7b4-4043-b402-6809e9429104\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb" Nov 01 06:26:17.909044 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:17.909010 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5dc4dc82-c7b4-4043-b402-6809e9429104-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb\" (UID: \"5dc4dc82-c7b4-4043-b402-6809e9429104\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb" Nov 01 06:26:17.968019 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:17.967984 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9sgq\" (UniqueName: \"kubernetes.io/projected/5dc4dc82-c7b4-4043-b402-6809e9429104-kube-api-access-z9sgq\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb\" (UID: \"5dc4dc82-c7b4-4043-b402-6809e9429104\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb" Nov 01 06:26:18.060849 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:18.060812 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb" Nov 01 06:26:18.440397 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:18.440363 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb"] Nov 01 06:26:18.440631 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:26:18.440608 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dc4dc82_c7b4_4043_b402_6809e9429104.slice/crio-a0ef748dae75eb4371f4611170921d84662bdabc6b533791070945f53811b2a3 WatchSource:0}: Error finding container a0ef748dae75eb4371f4611170921d84662bdabc6b533791070945f53811b2a3: Status 404 returned error can't find the container with id a0ef748dae75eb4371f4611170921d84662bdabc6b533791070945f53811b2a3 Nov 01 06:26:19.109049 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:19.109010 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb" event={"ID":"5dc4dc82-c7b4-4043-b402-6809e9429104","Type":"ContainerStarted","Data":"f4a738872483dac27229321a6110daaee599651dce6a9a3cdde1b1daba06f55f"} Nov 01 06:26:19.109541 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:19.109072 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb" event={"ID":"5dc4dc82-c7b4-4043-b402-6809e9429104","Type":"ContainerStarted","Data":"a0ef748dae75eb4371f4611170921d84662bdabc6b533791070945f53811b2a3"} Nov 01 06:26:19.109541 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:19.109102 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb" Nov 01 06:26:19.226749 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:19.226692 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb" podStartSLOduration=2.226676796 podStartE2EDuration="2.226676796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:26:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:26:19.226520325 +0000 UTC m=+2457.881637493" watchObservedRunningTime="2026-11-01 06:26:19.226676796 +0000 UTC m=+2457.881793964" Nov 01 06:26:22.848528 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:22.848492 5040 scope.go:122] "RemoveContainer" containerID="72c05979b7b2d6e3c17ee26d315c73f00ccb936b27af060b9403f95bcafeb80a" Nov 01 06:26:22.862268 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:22.862234 5040 scope.go:122] "RemoveContainer" containerID="50c0f92716c413352634aa4d7792601c030f4e7aa05e3fe94f95c328efd0d7af" Nov 01 06:26:24.617684 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:24.617362 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89596147-8d6s6"] Nov 01 06:26:24.647463 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:24.647424 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89596147-8d6s6" Nov 01 06:26:24.686994 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:24.682123 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf31556-92b7-491c-b685-987134f8282d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89596147-8d6s6\" (UID: \"8cf31556-92b7-491c-b685-987134f8282d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89596147-8d6s6" Nov 01 06:26:24.686994 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:24.682233 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf31556-92b7-491c-b685-987134f8282d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89596147-8d6s6\" (UID: \"8cf31556-92b7-491c-b685-987134f8282d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89596147-8d6s6" Nov 01 06:26:24.686994 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:24.682300 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf2ph\" (UniqueName: \"kubernetes.io/projected/8cf31556-92b7-491c-b685-987134f8282d-kube-api-access-pf2ph\") pod \"maas-api-revoke-sub-89596147-8d6s6\" (UID: \"8cf31556-92b7-491c-b685-987134f8282d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89596147-8d6s6" Nov 01 06:26:24.686994 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:24.684432 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89596147-8d6s6"] Nov 01 06:26:24.723558 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:24.723512 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:26:24.723558 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:24.723515 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:26:24.723827 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:24.723605 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:26:24.759157 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:24.759100 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:26:24.783439 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:24.783400 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf2ph\" (UniqueName: \"kubernetes.io/projected/8cf31556-92b7-491c-b685-987134f8282d-kube-api-access-pf2ph\") pod \"maas-api-revoke-sub-89596147-8d6s6\" (UID: \"8cf31556-92b7-491c-b685-987134f8282d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89596147-8d6s6" Nov 01 06:26:24.783839 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:24.783811 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf31556-92b7-491c-b685-987134f8282d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89596147-8d6s6\" (UID: \"8cf31556-92b7-491c-b685-987134f8282d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89596147-8d6s6" Nov 01 06:26:24.784086 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:24.784063 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf31556-92b7-491c-b685-987134f8282d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89596147-8d6s6\" (UID: \"8cf31556-92b7-491c-b685-987134f8282d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89596147-8d6s6" Nov 01 06:26:24.785003 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:24.784977 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf31556-92b7-491c-b685-987134f8282d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89596147-8d6s6\" (UID: \"8cf31556-92b7-491c-b685-987134f8282d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89596147-8d6s6" Nov 01 06:26:24.791135 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:24.787037 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf31556-92b7-491c-b685-987134f8282d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89596147-8d6s6\" (UID: \"8cf31556-92b7-491c-b685-987134f8282d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89596147-8d6s6" Nov 01 06:26:24.850193 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:24.850156 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf2ph\" (UniqueName: \"kubernetes.io/projected/8cf31556-92b7-491c-b685-987134f8282d-kube-api-access-pf2ph\") pod \"maas-api-revoke-sub-89596147-8d6s6\" (UID: \"8cf31556-92b7-491c-b685-987134f8282d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89596147-8d6s6" Nov 01 06:26:25.019503 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:25.018640 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89596147-8d6s6" Nov 01 06:26:25.095439 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:25.094354 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c93ac60-n7l49"] Nov 01 06:26:25.140197 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:25.140135 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c93ac60-n7l49"] Nov 01 06:26:25.558198 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:25.556622 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a8fc69b-5644-4cf5-930a-785de80cbc46" path="/var/lib/kubelet/pods/5a8fc69b-5644-4cf5-930a-785de80cbc46/volumes" Nov 01 06:26:25.558198 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:25.557091 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89596147-8d6s6"] Nov 01 06:26:26.201746 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:26.201703 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89596147-8d6s6" event={"ID":"8cf31556-92b7-491c-b685-987134f8282d","Type":"ContainerStarted","Data":"a52f97392d55ac682b59aab4fedeba9151d60d51cadbebeb0f9c995fd7a16271"} Nov 01 06:26:27.248779 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:27.248730 5040 generic.go:356] "Generic (PLEG): container finished" podID="8cf31556-92b7-491c-b685-987134f8282d" containerID="9205ca411e791d84fad2b3bbf56a46cbab6e5a7fbbbc81d699ab141870b5ed6b" exitCode=0 Nov 01 06:26:27.249492 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:27.249444 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89596147-8d6s6" event={"ID":"8cf31556-92b7-491c-b685-987134f8282d","Type":"ContainerDied","Data":"9205ca411e791d84fad2b3bbf56a46cbab6e5a7fbbbc81d699ab141870b5ed6b"} Nov 01 06:26:28.194503 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:28.193573 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e40-6t258"] Nov 01 06:26:28.235501 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:28.235445 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e40-6t258"] Nov 01 06:26:28.271982 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:28.271924 5040 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" podUID="57cf5e07-36de-48cf-b938-138484cea3a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:26:28.280801 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:28.280729 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" podUID="5daa03ca-824b-4799-a11b-4b3af1512c38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.123:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:26:28.385515 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:28.384871 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" podUID="21f54836-d1ed-4f87-8f26-7c2d939b2590" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Nov 01 06:26:28.427662 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:28.427597 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" podUID="75afe340-0b9c-4b4c-89ab-20d761e6b4f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 06:26:28.427987 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:28.427955 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" podUID="1878b6fb-f8fe-4f4f-8f4e-056c66fc206a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:26:28.481420 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:28.481303 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc" podUID="66fdf9da-4790-40c5-9434-052c7465d23e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.124:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 06:26:28.508746 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:28.508558 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" podUID="adbac784-4ded-4ca3-b985-3e59983fc637" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:26:28.522667 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:28.522608 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-brtbn" podUID="f94f4767-3f84-429a-bff5-38c4fa82deaa" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.41:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 06:26:28.583504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:28.583012 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" podUID="57cf5e07-36de-48cf-b938-138484cea3a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:26:28.946734 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:28.946701 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89596147-8d6s6" Nov 01 06:26:29.040073 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:29.039973 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cf31556-92b7-491c-b685-987134f8282d-kube-api-access-pf2ph\" (UniqueName: \"kubernetes.io/projected/8cf31556-92b7-491c-b685-987134f8282d-kube-api-access-pf2ph\") pod \"8cf31556-92b7-491c-b685-987134f8282d\" (UID: \"8cf31556-92b7-491c-b685-987134f8282d\") " Nov 01 06:26:29.040073 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:29.040032 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cf31556-92b7-491c-b685-987134f8282d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf31556-92b7-491c-b685-987134f8282d-maas-api-openshift-ca\") pod \"8cf31556-92b7-491c-b685-987134f8282d\" (UID: \"8cf31556-92b7-491c-b685-987134f8282d\") " Nov 01 06:26:29.040073 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:29.040071 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cf31556-92b7-491c-b685-987134f8282d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf31556-92b7-491c-b685-987134f8282d-maas-api-xks-ca\") pod \"8cf31556-92b7-491c-b685-987134f8282d\" (UID: \"8cf31556-92b7-491c-b685-987134f8282d\") " Nov 01 06:26:29.040591 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:29.040549 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cf31556-92b7-491c-b685-987134f8282d-maas-api-openshift-ca" pod "8cf31556-92b7-491c-b685-987134f8282d" (UID: "8cf31556-92b7-491c-b685-987134f8282d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:26:29.042260 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:29.042231 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cf31556-92b7-491c-b685-987134f8282d-kube-api-access-pf2ph" pod "8cf31556-92b7-491c-b685-987134f8282d" (UID: "8cf31556-92b7-491c-b685-987134f8282d"). InnerVolumeSpecName "kube-api-access-pf2ph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:26:29.042379 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:29.042304 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cf31556-92b7-491c-b685-987134f8282d-maas-api-xks-ca" pod "8cf31556-92b7-491c-b685-987134f8282d" (UID: "8cf31556-92b7-491c-b685-987134f8282d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:26:29.141236 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:29.141195 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf31556-92b7-491c-b685-987134f8282d-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:29.141236 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:29.141233 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf31556-92b7-491c-b685-987134f8282d-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:29.141236 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:29.141243 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf2ph\" (UniqueName: \"kubernetes.io/projected/8cf31556-92b7-491c-b685-987134f8282d-kube-api-access-pf2ph\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:29.292490 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:29.292390 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89596147-8d6s6" Nov 01 06:26:29.292490 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:29.292413 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89596147-8d6s6" event={"ID":"8cf31556-92b7-491c-b685-987134f8282d","Type":"ContainerDied","Data":"a52f97392d55ac682b59aab4fedeba9151d60d51cadbebeb0f9c995fd7a16271"} Nov 01 06:26:29.292930 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:29.292519 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a52f97392d55ac682b59aab4fedeba9151d60d51cadbebeb0f9c995fd7a16271" Nov 01 06:26:29.525745 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:29.525705 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fda7e954-48ee-4e6c-b04a-71df9940cb5a" path="/var/lib/kubelet/pods/fda7e954-48ee-4e6c-b04a-71df9940cb5a/volumes" Nov 01 06:26:30.122388 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:30.122344 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb" podUID="5dc4dc82-c7b4-4043-b402-6809e9429104" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:26:31.184069 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.184028 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv"] Nov 01 06:26:31.184608 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.184591 5040 cpu_manager.go:414] "containerMap: removing container" podUID="8cf31556-92b7-491c-b685-987134f8282d" containerName="revoke-keys" Nov 01 06:26:31.184673 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.184609 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cf31556-92b7-491c-b685-987134f8282d" containerName="revoke-keys" Nov 01 06:26:31.184743 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.184733 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cf31556-92b7-491c-b685-987134f8282d" containerName="revoke-keys" Nov 01 06:26:31.188297 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.188272 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:26:31.259500 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.243618 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ce346503-openshift-default-dockercfg-t52r8\"" Nov 01 06:26:31.277168 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.277112 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-credential-socket\") pod \"e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv\" (UID: \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\") " pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:26:31.277369 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.277170 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-istio-data\") pod \"e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv\" (UID: \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\") " pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:26:31.277369 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.277222 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-istiod-ca-cert\") pod \"e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv\" (UID: \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\") " pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:26:31.277369 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.277262 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-workload-certs\") pod \"e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv\" (UID: \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\") " pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:26:31.277369 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.277292 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-istio-envoy\") pod \"e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv\" (UID: \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\") " pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:26:31.277369 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.277333 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-istio-podinfo\") pod \"e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv\" (UID: \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\") " pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:26:31.277724 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.277406 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2cwx\" (UniqueName: \"kubernetes.io/projected/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-kube-api-access-c2cwx\") pod \"e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv\" (UID: \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\") " pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:26:31.277724 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.277533 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-istio-token\") pod \"e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv\" (UID: \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\") " pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:26:31.277724 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.277576 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-workload-socket\") pod \"e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv\" (UID: \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\") " pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:26:31.338398 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.338365 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv"] Nov 01 06:26:31.378906 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.378812 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-istio-token\") pod \"e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv\" (UID: \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\") " pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:26:31.379092 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.378990 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-workload-socket\") pod \"e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv\" (UID: \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\") " pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:26:31.379092 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.379078 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-credential-socket\") pod \"e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv\" (UID: \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\") " pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:26:31.379240 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.379109 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-istio-data\") pod \"e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv\" (UID: \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\") " pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:26:31.379240 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.379156 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-istiod-ca-cert\") pod \"e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv\" (UID: \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\") " pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:26:31.379240 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.379195 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-workload-certs\") pod \"e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv\" (UID: \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\") " pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:26:31.379240 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.379224 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-istio-envoy\") pod \"e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv\" (UID: \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\") " pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:26:31.379452 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.379264 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-istio-podinfo\") pod \"e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv\" (UID: \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\") " pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:26:31.379452 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.379341 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2cwx\" (UniqueName: \"kubernetes.io/projected/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-kube-api-access-c2cwx\") pod \"e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv\" (UID: \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\") " pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:26:31.380522 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.379651 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-credential-socket\") pod \"e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv\" (UID: \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\") " pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:26:31.380522 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.379682 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-workload-socket\") pod \"e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv\" (UID: \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\") " pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:26:31.380522 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.379869 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-workload-certs\") pod \"e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv\" (UID: \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\") " pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:26:31.380522 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.379969 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-istio-data\") pod \"e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv\" (UID: \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\") " pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:26:31.380522 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.380466 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-istiod-ca-cert\") pod \"e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv\" (UID: \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\") " pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:26:31.382883 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.382849 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-istio-envoy\") pod \"e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv\" (UID: \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\") " pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:26:31.389582 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.389549 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-istio-podinfo\") pod \"e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv\" (UID: \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\") " pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:26:31.440193 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.440161 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-istio-token\") pod \"e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv\" (UID: \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\") " pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:26:31.440412 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.440390 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2cwx\" (UniqueName: \"kubernetes.io/projected/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-kube-api-access-c2cwx\") pod \"e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv\" (UID: \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\") " pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:26:31.502833 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.502798 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:26:31.944831 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:26:31.944792 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a726a8_a06e_4f27_95e9_bca8b6b78f6d.slice/crio-3df9ed674a866a1ceaadefca0d243d87a46e70a12d5bf2c02a0d8d91e59c22dd WatchSource:0}: Error finding container 3df9ed674a866a1ceaadefca0d243d87a46e70a12d5bf2c02a0d8d91e59c22dd: Status 404 returned error can't find the container with id 3df9ed674a866a1ceaadefca0d243d87a46e70a12d5bf2c02a0d8d91e59c22dd Nov 01 06:26:31.947668 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.947623 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:26:31.947783 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.947720 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:26:31.947831 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.947782 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv"] Nov 01 06:26:31.947883 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:31.947789 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:26:32.354396 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:32.354353 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" event={"ID":"61a726a8-a06e-4f27-95e9-bca8b6b78f6d","Type":"ContainerStarted","Data":"c91c6d21f50b1bd24936f61867715b6115e4459b30f898aba28f2db276c632ef"} Nov 01 06:26:32.355040 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:32.354421 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" event={"ID":"61a726a8-a06e-4f27-95e9-bca8b6b78f6d","Type":"ContainerStarted","Data":"3df9ed674a866a1ceaadefca0d243d87a46e70a12d5bf2c02a0d8d91e59c22dd"} Nov 01 06:26:32.479724 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:32.479660 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" podStartSLOduration=1.479642544 podStartE2EDuration="1.479642544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:26:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:26:32.475511596 +0000 UTC m=+2471.130628764" watchObservedRunningTime="2026-11-01 06:26:32.479642544 +0000 UTC m=+2471.134759712" Nov 01 06:26:32.503234 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:32.503195 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:26:33.517530 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:33.517494 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:26:34.369517 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:34.369472 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:26:34.370528 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:34.370469 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:26:35.377822 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:35.377784 5040 generic.go:356] "Generic (PLEG): container finished" podID="068716fa-4615-4dbf-9c5f-41687865d5b1" containerID="44791eecd8e3223d742ac4c5c0b2f07057100ca4306ee7c0a39e839394079a15" exitCode=0 Nov 01 06:26:35.378288 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:35.377861 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" event={"ID":"068716fa-4615-4dbf-9c5f-41687865d5b1","Type":"ContainerDied","Data":"44791eecd8e3223d742ac4c5c0b2f07057100ca4306ee7c0a39e839394079a15"} Nov 01 06:26:35.378540 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:35.378511 5040 scope.go:122] "RemoveContainer" containerID="44791eecd8e3223d742ac4c5c0b2f07057100ca4306ee7c0a39e839394079a15" Nov 01 06:26:35.380217 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:35.380176 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:26:35.380346 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:35.380278 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:26:35.380346 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:35.380321 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:26:36.389177 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:36.389144 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" event={"ID":"068716fa-4615-4dbf-9c5f-41687865d5b1","Type":"ContainerStarted","Data":"3b141c3d9dc299bf597a555cd2cfdc2c9af0a7d6b9a3b4d1970e729214da666d"} Nov 01 06:26:36.948669 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:36.948632 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:26:36.948940 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:36.948914 5040 patch_prober.go:29] interesting pod/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 01 06:26:36.949014 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:36.948994 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" podUID="068716fa-4615-4dbf-9c5f-41687865d5b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 01 06:26:37.949184 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:37.949147 5040 patch_prober.go:29] interesting pod/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 01 06:26:37.949612 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:37.949205 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" podUID="068716fa-4615-4dbf-9c5f-41687865d5b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 01 06:26:38.948414 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:38.948376 5040 patch_prober.go:29] interesting pod/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 01 06:26:38.948641 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:38.948452 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" podUID="068716fa-4615-4dbf-9c5f-41687865d5b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 01 06:26:38.989323 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:38.989283 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb"] Nov 01 06:26:38.990647 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:38.990619 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb" podUID="5dc4dc82-c7b4-4043-b402-6809e9429104" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:26:39.948221 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:39.948190 5040 patch_prober.go:29] interesting pod/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 01 06:26:39.948415 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:39.948248 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" podUID="068716fa-4615-4dbf-9c5f-41687865d5b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 01 06:26:39.974708 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:26:39.974667 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:26:40.949133 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:40.949089 5040 patch_prober.go:29] interesting pod/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 01 06:26:40.949702 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:40.949158 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" podUID="068716fa-4615-4dbf-9c5f-41687865d5b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 01 06:26:41.952830 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:41.952792 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:26:41.953236 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:41.953200 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:26:41.953884 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:41.953855 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:26:43.990427 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:43.990359 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb" podUID="5dc4dc82-c7b4-4043-b402-6809e9429104" containerName="payload-processing" containerID="cri-o://f4a738872483dac27229321a6110daaee599651dce6a9a3cdde1b1daba06f55f" gracePeriod=25 Nov 01 06:26:44.255819 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:44.255790 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb" Nov 01 06:26:44.365214 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:44.365175 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dc4dc82-c7b4-4043-b402-6809e9429104-kube-api-access-z9sgq\" (UniqueName: \"kubernetes.io/projected/5dc4dc82-c7b4-4043-b402-6809e9429104-kube-api-access-z9sgq\") pod \"5dc4dc82-c7b4-4043-b402-6809e9429104\" (UID: \"5dc4dc82-c7b4-4043-b402-6809e9429104\") " Nov 01 06:26:44.365401 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:44.365250 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dc4dc82-c7b4-4043-b402-6809e9429104-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc4dc82-c7b4-4043-b402-6809e9429104-plugins-config-volume\") pod \"5dc4dc82-c7b4-4043-b402-6809e9429104\" (UID: \"5dc4dc82-c7b4-4043-b402-6809e9429104\") " Nov 01 06:26:44.365401 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:44.365308 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dc4dc82-c7b4-4043-b402-6809e9429104-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5dc4dc82-c7b4-4043-b402-6809e9429104-routing-overlay-volume\") pod \"5dc4dc82-c7b4-4043-b402-6809e9429104\" (UID: \"5dc4dc82-c7b4-4043-b402-6809e9429104\") " Nov 01 06:26:44.365527 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:44.365405 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dc4dc82-c7b4-4043-b402-6809e9429104-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5dc4dc82-c7b4-4043-b402-6809e9429104-provider-credentials\") pod \"5dc4dc82-c7b4-4043-b402-6809e9429104\" (UID: \"5dc4dc82-c7b4-4043-b402-6809e9429104\") " Nov 01 06:26:44.365762 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:44.365726 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dc4dc82-c7b4-4043-b402-6809e9429104-routing-overlay-volume" pod "5dc4dc82-c7b4-4043-b402-6809e9429104" (UID: "5dc4dc82-c7b4-4043-b402-6809e9429104"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:26:44.365762 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:44.365736 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dc4dc82-c7b4-4043-b402-6809e9429104-plugins-config-volume" pod "5dc4dc82-c7b4-4043-b402-6809e9429104" (UID: "5dc4dc82-c7b4-4043-b402-6809e9429104"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:26:44.365924 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:44.365847 5040 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc4dc82-c7b4-4043-b402-6809e9429104-plugins-config-volume\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:44.365924 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:44.365859 5040 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5dc4dc82-c7b4-4043-b402-6809e9429104-routing-overlay-volume\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:44.367547 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:44.367522 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc4dc82-c7b4-4043-b402-6809e9429104-provider-credentials" pod "5dc4dc82-c7b4-4043-b402-6809e9429104" (UID: "5dc4dc82-c7b4-4043-b402-6809e9429104"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:26:44.367645 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:44.367625 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc4dc82-c7b4-4043-b402-6809e9429104-kube-api-access-z9sgq" pod "5dc4dc82-c7b4-4043-b402-6809e9429104" (UID: "5dc4dc82-c7b4-4043-b402-6809e9429104"). InnerVolumeSpecName "kube-api-access-z9sgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:26:44.467276 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:44.467235 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9sgq\" (UniqueName: \"kubernetes.io/projected/5dc4dc82-c7b4-4043-b402-6809e9429104-kube-api-access-z9sgq\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:44.467276 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:44.467274 5040 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5dc4dc82-c7b4-4043-b402-6809e9429104-provider-credentials\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:44.560439 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:44.560350 5040 generic.go:356] "Generic (PLEG): container finished" podID="5dc4dc82-c7b4-4043-b402-6809e9429104" containerID="f4a738872483dac27229321a6110daaee599651dce6a9a3cdde1b1daba06f55f" exitCode=0 Nov 01 06:26:44.560439 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:44.560405 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb" Nov 01 06:26:44.560669 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:44.560432 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb" event={"ID":"5dc4dc82-c7b4-4043-b402-6809e9429104","Type":"ContainerDied","Data":"f4a738872483dac27229321a6110daaee599651dce6a9a3cdde1b1daba06f55f"} Nov 01 06:26:44.560669 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:44.560503 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb" event={"ID":"5dc4dc82-c7b4-4043-b402-6809e9429104","Type":"ContainerDied","Data":"a0ef748dae75eb4371f4611170921d84662bdabc6b533791070945f53811b2a3"} Nov 01 06:26:44.560669 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:44.560528 5040 scope.go:122] "RemoveContainer" containerID="f4a738872483dac27229321a6110daaee599651dce6a9a3cdde1b1daba06f55f" Nov 01 06:26:44.574949 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:44.574852 5040 scope.go:122] "RemoveContainer" containerID="f4a738872483dac27229321a6110daaee599651dce6a9a3cdde1b1daba06f55f" Nov 01 06:26:44.575442 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:26:44.575407 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4a738872483dac27229321a6110daaee599651dce6a9a3cdde1b1daba06f55f\": container with ID starting with f4a738872483dac27229321a6110daaee599651dce6a9a3cdde1b1daba06f55f not found: ID does not exist" containerID="f4a738872483dac27229321a6110daaee599651dce6a9a3cdde1b1daba06f55f" Nov 01 06:26:44.575597 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:44.575445 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4a738872483dac27229321a6110daaee599651dce6a9a3cdde1b1daba06f55f"} err="failed to get container status \"f4a738872483dac27229321a6110daaee599651dce6a9a3cdde1b1daba06f55f\": rpc error: code = NotFound desc = could not find container \"f4a738872483dac27229321a6110daaee599651dce6a9a3cdde1b1daba06f55f\": container with ID starting with f4a738872483dac27229321a6110daaee599651dce6a9a3cdde1b1daba06f55f not found: ID does not exist" Nov 01 06:26:44.731955 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:44.731916 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb"] Nov 01 06:26:44.747369 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:44.747335 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbd98mb"] Nov 01 06:26:45.515183 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:45.515145 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dc4dc82-c7b4-4043-b402-6809e9429104" path="/var/lib/kubelet/pods/5dc4dc82-c7b4-4043-b402-6809e9429104/volumes" Nov 01 06:26:59.952207 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:59.952121 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx"] Nov 01 06:26:59.953071 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:26:59.953037 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" podUID="b0f9327e-b3f9-4af9-b9c9-506a8859a46f" containerName="istio-proxy" containerID="cri-o://46ed39363028c516cda78e8d043de47223dcb31f369ab51217d74c38a439aca8" gracePeriod=30 Nov 01 06:27:03.350511 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:03.348619 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" podUID="5daa03ca-824b-4799-a11b-4b3af1512c38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.123:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 06:27:03.423981 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:03.423933 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" podUID="1878b6fb-f8fe-4f4f-8f4e-056c66fc206a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Nov 01 06:27:03.501619 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:03.501565 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" podUID="adbac784-4ded-4ca3-b985-3e59983fc637" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:27:05.243838 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.243758 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:27:05.319464 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.319420 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-workload-socket\") pod \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\" (UID: \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\") " Nov 01 06:27:05.319683 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.319526 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-workload-certs\") pod \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\" (UID: \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\") " Nov 01 06:27:05.319683 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.319580 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-istio-token\" (UniqueName: \"kubernetes.io/projected/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-istio-token\") pod \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\" (UID: \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\") " Nov 01 06:27:05.319683 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.319610 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-kube-api-access-47csm\" (UniqueName: \"kubernetes.io/projected/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-kube-api-access-47csm\") pod \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\" (UID: \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\") " Nov 01 06:27:05.319683 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.319661 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-istio-envoy\") pod \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\" (UID: \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\") " Nov 01 06:27:05.319915 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.319699 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-istio-data\") pod \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\" (UID: \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\") " Nov 01 06:27:05.319915 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.319781 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-workload-socket" pod "b0f9327e-b3f9-4af9-b9c9-506a8859a46f" (UID: "b0f9327e-b3f9-4af9-b9c9-506a8859a46f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:27:05.319915 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.319794 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-workload-certs" pod "b0f9327e-b3f9-4af9-b9c9-506a8859a46f" (UID: "b0f9327e-b3f9-4af9-b9c9-506a8859a46f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:27:05.319915 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.319816 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-istiod-ca-cert\") pod \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\" (UID: \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\") " Nov 01 06:27:05.319915 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.319876 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-istio-podinfo\") pod \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\" (UID: \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\") " Nov 01 06:27:05.320201 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.319917 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-credential-socket\") pod \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\" (UID: \"b0f9327e-b3f9-4af9-b9c9-506a8859a46f\") " Nov 01 06:27:05.320201 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.320056 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-istio-data" pod "b0f9327e-b3f9-4af9-b9c9-506a8859a46f" (UID: "b0f9327e-b3f9-4af9-b9c9-506a8859a46f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:27:05.320315 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.320298 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-credential-socket" pod "b0f9327e-b3f9-4af9-b9c9-506a8859a46f" (UID: "b0f9327e-b3f9-4af9-b9c9-506a8859a46f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:27:05.320370 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.320305 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-istiod-ca-cert" pod "b0f9327e-b3f9-4af9-b9c9-506a8859a46f" (UID: "b0f9327e-b3f9-4af9-b9c9-506a8859a46f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:27:05.320434 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.320397 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:05.320434 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.320418 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:05.320561 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.320433 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:05.320561 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.320448 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:05.322076 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.322038 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-istio-envoy" pod "b0f9327e-b3f9-4af9-b9c9-506a8859a46f" (UID: "b0f9327e-b3f9-4af9-b9c9-506a8859a46f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:27:05.322076 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.322058 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-istio-token" pod "b0f9327e-b3f9-4af9-b9c9-506a8859a46f" (UID: "b0f9327e-b3f9-4af9-b9c9-506a8859a46f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:27:05.322300 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.322078 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-kube-api-access-47csm" pod "b0f9327e-b3f9-4af9-b9c9-506a8859a46f" (UID: "b0f9327e-b3f9-4af9-b9c9-506a8859a46f"). InnerVolumeSpecName "kube-api-access-47csm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:27:05.322540 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.322512 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-istio-podinfo" pod "b0f9327e-b3f9-4af9-b9c9-506a8859a46f" (UID: "b0f9327e-b3f9-4af9-b9c9-506a8859a46f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:27:05.421906 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.421792 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:05.421906 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.421826 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:05.421906 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.421840 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47csm\" (UniqueName: \"kubernetes.io/projected/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-kube-api-access-47csm\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:05.421906 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.421855 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:05.421906 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.421868 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0f9327e-b3f9-4af9-b9c9-506a8859a46f-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:05.855058 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.854966 5040 generic.go:356] "Generic (PLEG): container finished" podID="b0f9327e-b3f9-4af9-b9c9-506a8859a46f" containerID="46ed39363028c516cda78e8d043de47223dcb31f369ab51217d74c38a439aca8" exitCode=0 Nov 01 06:27:05.855058 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.855015 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" Nov 01 06:27:05.855058 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.855045 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" event={"ID":"b0f9327e-b3f9-4af9-b9c9-506a8859a46f","Type":"ContainerDied","Data":"46ed39363028c516cda78e8d043de47223dcb31f369ab51217d74c38a439aca8"} Nov 01 06:27:05.855275 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.855105 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx" event={"ID":"b0f9327e-b3f9-4af9-b9c9-506a8859a46f","Type":"ContainerDied","Data":"ec8b712a7fec5450a9abbe6838984aa38a69946a9b40b823bf11f5a440abd241"} Nov 01 06:27:05.855275 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.855138 5040 scope.go:122] "RemoveContainer" containerID="46ed39363028c516cda78e8d043de47223dcb31f369ab51217d74c38a439aca8" Nov 01 06:27:05.866608 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.866580 5040 scope.go:122] "RemoveContainer" containerID="0ebf75d2da3392bd93380e3ce2354794bc3c53dd1aa42ab3274ef8bac9db512d" Nov 01 06:27:05.878165 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.878140 5040 scope.go:122] "RemoveContainer" containerID="46ed39363028c516cda78e8d043de47223dcb31f369ab51217d74c38a439aca8" Nov 01 06:27:05.878564 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:27:05.878535 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46ed39363028c516cda78e8d043de47223dcb31f369ab51217d74c38a439aca8\": container with ID starting with 46ed39363028c516cda78e8d043de47223dcb31f369ab51217d74c38a439aca8 not found: ID does not exist" containerID="46ed39363028c516cda78e8d043de47223dcb31f369ab51217d74c38a439aca8" Nov 01 06:27:05.878697 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.878574 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46ed39363028c516cda78e8d043de47223dcb31f369ab51217d74c38a439aca8"} err="failed to get container status \"46ed39363028c516cda78e8d043de47223dcb31f369ab51217d74c38a439aca8\": rpc error: code = NotFound desc = could not find container \"46ed39363028c516cda78e8d043de47223dcb31f369ab51217d74c38a439aca8\": container with ID starting with 46ed39363028c516cda78e8d043de47223dcb31f369ab51217d74c38a439aca8 not found: ID does not exist" Nov 01 06:27:05.878697 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.878602 5040 scope.go:122] "RemoveContainer" containerID="0ebf75d2da3392bd93380e3ce2354794bc3c53dd1aa42ab3274ef8bac9db512d" Nov 01 06:27:05.878953 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:27:05.878931 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ebf75d2da3392bd93380e3ce2354794bc3c53dd1aa42ab3274ef8bac9db512d\": container with ID starting with 0ebf75d2da3392bd93380e3ce2354794bc3c53dd1aa42ab3274ef8bac9db512d not found: ID does not exist" containerID="0ebf75d2da3392bd93380e3ce2354794bc3c53dd1aa42ab3274ef8bac9db512d" Nov 01 06:27:05.879003 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:05.878960 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ebf75d2da3392bd93380e3ce2354794bc3c53dd1aa42ab3274ef8bac9db512d"} err="failed to get container status \"0ebf75d2da3392bd93380e3ce2354794bc3c53dd1aa42ab3274ef8bac9db512d\": rpc error: code = NotFound desc = could not find container \"0ebf75d2da3392bd93380e3ce2354794bc3c53dd1aa42ab3274ef8bac9db512d\": container with ID starting with 0ebf75d2da3392bd93380e3ce2354794bc3c53dd1aa42ab3274ef8bac9db512d not found: ID does not exist" Nov 01 06:27:06.024390 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:06.024347 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx"] Nov 01 06:27:06.075190 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:06.075155 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c7e551a7-openshift-default-7fdc8c84f7-cwnvx"] Nov 01 06:27:07.514524 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:07.514471 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0f9327e-b3f9-4af9-b9c9-506a8859a46f" path="/var/lib/kubelet/pods/b0f9327e-b3f9-4af9-b9c9-506a8859a46f/volumes" Nov 01 06:27:09.996784 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:27:09.996744 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:27:22.889161 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:22.889115 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a802d7ec-zrw5m"] Nov 01 06:27:22.889619 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:22.889606 5040 cpu_manager.go:414] "containerMap: removing container" podUID="b0f9327e-b3f9-4af9-b9c9-506a8859a46f" containerName="istio-proxy" Nov 01 06:27:22.889668 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:22.889620 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0f9327e-b3f9-4af9-b9c9-506a8859a46f" containerName="istio-proxy" Nov 01 06:27:22.889668 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:22.889641 5040 cpu_manager.go:414] "containerMap: removing container" podUID="b0f9327e-b3f9-4af9-b9c9-506a8859a46f" containerName="istio-proxy" Nov 01 06:27:22.889668 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:22.889646 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0f9327e-b3f9-4af9-b9c9-506a8859a46f" containerName="istio-proxy" Nov 01 06:27:22.889668 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:22.889652 5040 cpu_manager.go:414] "containerMap: removing container" podUID="5dc4dc82-c7b4-4043-b402-6809e9429104" containerName="payload-processing" Nov 01 06:27:22.889668 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:22.889656 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dc4dc82-c7b4-4043-b402-6809e9429104" containerName="payload-processing" Nov 01 06:27:22.889826 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:22.889758 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0f9327e-b3f9-4af9-b9c9-506a8859a46f" containerName="istio-proxy" Nov 01 06:27:22.889826 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:22.889772 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0f9327e-b3f9-4af9-b9c9-506a8859a46f" containerName="istio-proxy" Nov 01 06:27:22.889826 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:22.889782 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dc4dc82-c7b4-4043-b402-6809e9429104" containerName="payload-processing" Nov 01 06:27:22.893194 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:22.893175 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a802d7ec-zrw5m" Nov 01 06:27:22.904713 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:22.904682 5040 scope.go:122] "RemoveContainer" containerID="08b3e565b35a53448663ddbec4921c43132e5c4284371251526167d488751a58" Nov 01 06:27:22.950797 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:22.950767 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a802d7ec-zrw5m"] Nov 01 06:27:22.996936 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:22.996900 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:27:22.997148 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:22.996994 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:27:22.997148 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:22.997016 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:27:22.997148 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:22.996993 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:27:23.017818 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:23.017788 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cffa541-7c5c-41c9-8c14-4a97a9375234-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a802d7ec-zrw5m\" (UID: \"5cffa541-7c5c-41c9-8c14-4a97a9375234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a802d7ec-zrw5m" Nov 01 06:27:23.017988 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:23.017824 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbskz\" (UniqueName: \"kubernetes.io/projected/5cffa541-7c5c-41c9-8c14-4a97a9375234-kube-api-access-pbskz\") pod \"maas-api-revoke-sub-a802d7ec-zrw5m\" (UID: \"5cffa541-7c5c-41c9-8c14-4a97a9375234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a802d7ec-zrw5m" Nov 01 06:27:23.017988 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:23.017974 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cffa541-7c5c-41c9-8c14-4a97a9375234-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a802d7ec-zrw5m\" (UID: \"5cffa541-7c5c-41c9-8c14-4a97a9375234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a802d7ec-zrw5m" Nov 01 06:27:23.118576 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:23.118538 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cffa541-7c5c-41c9-8c14-4a97a9375234-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a802d7ec-zrw5m\" (UID: \"5cffa541-7c5c-41c9-8c14-4a97a9375234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a802d7ec-zrw5m" Nov 01 06:27:23.118795 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:23.118636 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cffa541-7c5c-41c9-8c14-4a97a9375234-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a802d7ec-zrw5m\" (UID: \"5cffa541-7c5c-41c9-8c14-4a97a9375234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a802d7ec-zrw5m" Nov 01 06:27:23.118795 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:23.118658 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbskz\" (UniqueName: \"kubernetes.io/projected/5cffa541-7c5c-41c9-8c14-4a97a9375234-kube-api-access-pbskz\") pod \"maas-api-revoke-sub-a802d7ec-zrw5m\" (UID: \"5cffa541-7c5c-41c9-8c14-4a97a9375234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a802d7ec-zrw5m" Nov 01 06:27:23.119273 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:23.119248 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cffa541-7c5c-41c9-8c14-4a97a9375234-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a802d7ec-zrw5m\" (UID: \"5cffa541-7c5c-41c9-8c14-4a97a9375234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a802d7ec-zrw5m" Nov 01 06:27:23.121100 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:23.121073 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cffa541-7c5c-41c9-8c14-4a97a9375234-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a802d7ec-zrw5m\" (UID: \"5cffa541-7c5c-41c9-8c14-4a97a9375234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a802d7ec-zrw5m" Nov 01 06:27:23.180183 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:23.180103 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbskz\" (UniqueName: \"kubernetes.io/projected/5cffa541-7c5c-41c9-8c14-4a97a9375234-kube-api-access-pbskz\") pod \"maas-api-revoke-sub-a802d7ec-zrw5m\" (UID: \"5cffa541-7c5c-41c9-8c14-4a97a9375234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a802d7ec-zrw5m" Nov 01 06:27:23.208665 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:23.208622 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a802d7ec-zrw5m" Nov 01 06:27:23.588868 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:23.588835 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a802d7ec-zrw5m"] Nov 01 06:27:23.589144 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:27:23.589120 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cffa541_7c5c_41c9_8c14_4a97a9375234.slice/crio-d584ff1959df13e911dd04afc90d8c26802908e1899e786d96074a9f31631329 WatchSource:0}: Error finding container d584ff1959df13e911dd04afc90d8c26802908e1899e786d96074a9f31631329: Status 404 returned error can't find the container with id d584ff1959df13e911dd04afc90d8c26802908e1899e786d96074a9f31631329 Nov 01 06:27:23.986142 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:23.986105 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a802d7ec-zrw5m" event={"ID":"5cffa541-7c5c-41c9-8c14-4a97a9375234","Type":"ContainerStarted","Data":"61e90ae756846959eb12b6c79257e7f80ca03ebb46e3e02c4e94d0b6b7dfee96"} Nov 01 06:27:23.986614 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:23.986159 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a802d7ec-zrw5m" event={"ID":"5cffa541-7c5c-41c9-8c14-4a97a9375234","Type":"ContainerStarted","Data":"d584ff1959df13e911dd04afc90d8c26802908e1899e786d96074a9f31631329"} Nov 01 06:27:24.104065 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:24.104009 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a802d7ec-zrw5m" podStartSLOduration=2.103994738 podStartE2EDuration="2.103994738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:27:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:27:24.102418432 +0000 UTC m=+2522.757535598" watchObservedRunningTime="2026-11-01 06:27:24.103994738 +0000 UTC m=+2522.759111905" Nov 01 06:27:24.994664 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:24.994626 5040 generic.go:356] "Generic (PLEG): container finished" podID="5cffa541-7c5c-41c9-8c14-4a97a9375234" containerID="61e90ae756846959eb12b6c79257e7f80ca03ebb46e3e02c4e94d0b6b7dfee96" exitCode=0 Nov 01 06:27:24.995072 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:24.994715 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a802d7ec-zrw5m" event={"ID":"5cffa541-7c5c-41c9-8c14-4a97a9375234","Type":"ContainerDied","Data":"61e90ae756846959eb12b6c79257e7f80ca03ebb46e3e02c4e94d0b6b7dfee96"} Nov 01 06:27:26.151505 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:26.151452 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a802d7ec-zrw5m" Nov 01 06:27:26.256119 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:26.256021 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cffa541-7c5c-41c9-8c14-4a97a9375234-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cffa541-7c5c-41c9-8c14-4a97a9375234-maas-api-xks-ca\") pod \"5cffa541-7c5c-41c9-8c14-4a97a9375234\" (UID: \"5cffa541-7c5c-41c9-8c14-4a97a9375234\") " Nov 01 06:27:26.256310 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:26.256124 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cffa541-7c5c-41c9-8c14-4a97a9375234-kube-api-access-pbskz\" (UniqueName: \"kubernetes.io/projected/5cffa541-7c5c-41c9-8c14-4a97a9375234-kube-api-access-pbskz\") pod \"5cffa541-7c5c-41c9-8c14-4a97a9375234\" (UID: \"5cffa541-7c5c-41c9-8c14-4a97a9375234\") " Nov 01 06:27:26.256310 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:26.256191 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cffa541-7c5c-41c9-8c14-4a97a9375234-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cffa541-7c5c-41c9-8c14-4a97a9375234-maas-api-openshift-ca\") pod \"5cffa541-7c5c-41c9-8c14-4a97a9375234\" (UID: \"5cffa541-7c5c-41c9-8c14-4a97a9375234\") " Nov 01 06:27:26.256599 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:26.256573 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cffa541-7c5c-41c9-8c14-4a97a9375234-maas-api-openshift-ca" pod "5cffa541-7c5c-41c9-8c14-4a97a9375234" (UID: "5cffa541-7c5c-41c9-8c14-4a97a9375234"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:27:26.258225 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:26.258201 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cffa541-7c5c-41c9-8c14-4a97a9375234-maas-api-xks-ca" pod "5cffa541-7c5c-41c9-8c14-4a97a9375234" (UID: "5cffa541-7c5c-41c9-8c14-4a97a9375234"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:27:26.258298 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:26.258221 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cffa541-7c5c-41c9-8c14-4a97a9375234-kube-api-access-pbskz" pod "5cffa541-7c5c-41c9-8c14-4a97a9375234" (UID: "5cffa541-7c5c-41c9-8c14-4a97a9375234"). InnerVolumeSpecName "kube-api-access-pbskz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:27:26.357612 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:26.357571 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cffa541-7c5c-41c9-8c14-4a97a9375234-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:26.357612 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:26.357606 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbskz\" (UniqueName: \"kubernetes.io/projected/5cffa541-7c5c-41c9-8c14-4a97a9375234-kube-api-access-pbskz\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:26.357612 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:26.357620 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cffa541-7c5c-41c9-8c14-4a97a9375234-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:27.010063 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:27.010023 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a802d7ec-zrw5m" event={"ID":"5cffa541-7c5c-41c9-8c14-4a97a9375234","Type":"ContainerDied","Data":"d584ff1959df13e911dd04afc90d8c26802908e1899e786d96074a9f31631329"} Nov 01 06:27:27.010063 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:27.010050 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a802d7ec-zrw5m" Nov 01 06:27:27.010063 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:27.010071 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d584ff1959df13e911dd04afc90d8c26802908e1899e786d96074a9f31631329" Nov 01 06:27:27.771698 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:27.771659 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk"] Nov 01 06:27:27.772239 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:27.772156 5040 cpu_manager.go:414] "containerMap: removing container" podUID="5cffa541-7c5c-41c9-8c14-4a97a9375234" containerName="revoke-keys" Nov 01 06:27:27.772239 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:27.772170 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cffa541-7c5c-41c9-8c14-4a97a9375234" containerName="revoke-keys" Nov 01 06:27:27.772367 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:27.772286 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cffa541-7c5c-41c9-8c14-4a97a9375234" containerName="revoke-keys" Nov 01 06:27:27.775797 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:27.775774 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:27:27.836270 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:27.836121 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-745a18-openshift-default-dockercfg-vqzv5\"" Nov 01 06:27:27.880811 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:27.880763 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p24qh\" (UniqueName: \"kubernetes.io/projected/f2e934b9-b834-487b-9a7b-08f3047c49c6-kube-api-access-p24qh\") pod \"e2e-api-b-745a18-openshift-default-9d949867f-kq5zk\" (UID: \"f2e934b9-b834-487b-9a7b-08f3047c49c6\") " pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:27:27.881018 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:27.880867 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2e934b9-b834-487b-9a7b-08f3047c49c6-istio-token\") pod \"e2e-api-b-745a18-openshift-default-9d949867f-kq5zk\" (UID: \"f2e934b9-b834-487b-9a7b-08f3047c49c6\") " pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:27:27.881018 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:27.880913 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2e934b9-b834-487b-9a7b-08f3047c49c6-istio-podinfo\") pod \"e2e-api-b-745a18-openshift-default-9d949867f-kq5zk\" (UID: \"f2e934b9-b834-487b-9a7b-08f3047c49c6\") " pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:27:27.881152 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:27.881012 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2e934b9-b834-487b-9a7b-08f3047c49c6-istiod-ca-cert\") pod \"e2e-api-b-745a18-openshift-default-9d949867f-kq5zk\" (UID: \"f2e934b9-b834-487b-9a7b-08f3047c49c6\") " pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:27:27.881152 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:27.881085 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2e934b9-b834-487b-9a7b-08f3047c49c6-credential-socket\") pod \"e2e-api-b-745a18-openshift-default-9d949867f-kq5zk\" (UID: \"f2e934b9-b834-487b-9a7b-08f3047c49c6\") " pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:27:27.881152 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:27.881144 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2e934b9-b834-487b-9a7b-08f3047c49c6-istio-data\") pod \"e2e-api-b-745a18-openshift-default-9d949867f-kq5zk\" (UID: \"f2e934b9-b834-487b-9a7b-08f3047c49c6\") " pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:27:27.881335 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:27.881190 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2e934b9-b834-487b-9a7b-08f3047c49c6-istio-envoy\") pod \"e2e-api-b-745a18-openshift-default-9d949867f-kq5zk\" (UID: \"f2e934b9-b834-487b-9a7b-08f3047c49c6\") " pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:27:27.881335 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:27.881250 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2e934b9-b834-487b-9a7b-08f3047c49c6-workload-socket\") pod \"e2e-api-b-745a18-openshift-default-9d949867f-kq5zk\" (UID: \"f2e934b9-b834-487b-9a7b-08f3047c49c6\") " pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:27:27.881335 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:27.881323 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2e934b9-b834-487b-9a7b-08f3047c49c6-workload-certs\") pod \"e2e-api-b-745a18-openshift-default-9d949867f-kq5zk\" (UID: \"f2e934b9-b834-487b-9a7b-08f3047c49c6\") " pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:27:27.894059 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:27.894010 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk"] Nov 01 06:27:27.982623 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:27.982587 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2e934b9-b834-487b-9a7b-08f3047c49c6-workload-socket\") pod \"e2e-api-b-745a18-openshift-default-9d949867f-kq5zk\" (UID: \"f2e934b9-b834-487b-9a7b-08f3047c49c6\") " pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:27:27.982828 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:27.982640 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2e934b9-b834-487b-9a7b-08f3047c49c6-workload-certs\") pod \"e2e-api-b-745a18-openshift-default-9d949867f-kq5zk\" (UID: \"f2e934b9-b834-487b-9a7b-08f3047c49c6\") " pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:27:27.982828 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:27.982729 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p24qh\" (UniqueName: \"kubernetes.io/projected/f2e934b9-b834-487b-9a7b-08f3047c49c6-kube-api-access-p24qh\") pod \"e2e-api-b-745a18-openshift-default-9d949867f-kq5zk\" (UID: \"f2e934b9-b834-487b-9a7b-08f3047c49c6\") " pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:27:27.982828 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:27.982774 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2e934b9-b834-487b-9a7b-08f3047c49c6-istio-token\") pod \"e2e-api-b-745a18-openshift-default-9d949867f-kq5zk\" (UID: \"f2e934b9-b834-487b-9a7b-08f3047c49c6\") " pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:27:27.982828 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:27.982801 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2e934b9-b834-487b-9a7b-08f3047c49c6-istio-podinfo\") pod \"e2e-api-b-745a18-openshift-default-9d949867f-kq5zk\" (UID: \"f2e934b9-b834-487b-9a7b-08f3047c49c6\") " pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:27:27.983073 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:27.982887 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2e934b9-b834-487b-9a7b-08f3047c49c6-istiod-ca-cert\") pod \"e2e-api-b-745a18-openshift-default-9d949867f-kq5zk\" (UID: \"f2e934b9-b834-487b-9a7b-08f3047c49c6\") " pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:27:27.983073 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:27.982940 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2e934b9-b834-487b-9a7b-08f3047c49c6-credential-socket\") pod \"e2e-api-b-745a18-openshift-default-9d949867f-kq5zk\" (UID: \"f2e934b9-b834-487b-9a7b-08f3047c49c6\") " pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:27:27.983073 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:27.982980 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2e934b9-b834-487b-9a7b-08f3047c49c6-istio-data\") pod \"e2e-api-b-745a18-openshift-default-9d949867f-kq5zk\" (UID: \"f2e934b9-b834-487b-9a7b-08f3047c49c6\") " pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:27:27.983073 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:27.983021 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2e934b9-b834-487b-9a7b-08f3047c49c6-istio-envoy\") pod \"e2e-api-b-745a18-openshift-default-9d949867f-kq5zk\" (UID: \"f2e934b9-b834-487b-9a7b-08f3047c49c6\") " pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:27:27.983312 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:27.983104 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2e934b9-b834-487b-9a7b-08f3047c49c6-workload-socket\") pod \"e2e-api-b-745a18-openshift-default-9d949867f-kq5zk\" (UID: \"f2e934b9-b834-487b-9a7b-08f3047c49c6\") " pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:27:27.983312 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:27.983164 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2e934b9-b834-487b-9a7b-08f3047c49c6-workload-certs\") pod \"e2e-api-b-745a18-openshift-default-9d949867f-kq5zk\" (UID: \"f2e934b9-b834-487b-9a7b-08f3047c49c6\") " pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:27:27.984060 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:27.984035 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2e934b9-b834-487b-9a7b-08f3047c49c6-istiod-ca-cert\") pod \"e2e-api-b-745a18-openshift-default-9d949867f-kq5zk\" (UID: \"f2e934b9-b834-487b-9a7b-08f3047c49c6\") " pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:27:27.984288 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:27.984268 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2e934b9-b834-487b-9a7b-08f3047c49c6-credential-socket\") pod \"e2e-api-b-745a18-openshift-default-9d949867f-kq5zk\" (UID: \"f2e934b9-b834-487b-9a7b-08f3047c49c6\") " pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:27:27.984809 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:27.984756 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2e934b9-b834-487b-9a7b-08f3047c49c6-istio-data\") pod \"e2e-api-b-745a18-openshift-default-9d949867f-kq5zk\" (UID: \"f2e934b9-b834-487b-9a7b-08f3047c49c6\") " pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:27:27.986563 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:27.986540 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2e934b9-b834-487b-9a7b-08f3047c49c6-istio-podinfo\") pod \"e2e-api-b-745a18-openshift-default-9d949867f-kq5zk\" (UID: \"f2e934b9-b834-487b-9a7b-08f3047c49c6\") " pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:27:27.986666 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:27.986576 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2e934b9-b834-487b-9a7b-08f3047c49c6-istio-envoy\") pod \"e2e-api-b-745a18-openshift-default-9d949867f-kq5zk\" (UID: \"f2e934b9-b834-487b-9a7b-08f3047c49c6\") " pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:27:28.045442 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:28.045364 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p24qh\" (UniqueName: \"kubernetes.io/projected/f2e934b9-b834-487b-9a7b-08f3047c49c6-kube-api-access-p24qh\") pod \"e2e-api-b-745a18-openshift-default-9d949867f-kq5zk\" (UID: \"f2e934b9-b834-487b-9a7b-08f3047c49c6\") " pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:27:28.045638 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:28.045468 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2e934b9-b834-487b-9a7b-08f3047c49c6-istio-token\") pod \"e2e-api-b-745a18-openshift-default-9d949867f-kq5zk\" (UID: \"f2e934b9-b834-487b-9a7b-08f3047c49c6\") " pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:27:28.089839 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:28.089772 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:27:28.527904 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:27:28.527867 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2e934b9_b834_487b_9a7b_08f3047c49c6.slice/crio-c185d891fff98e841d23289b330b89ec62882f4a058a6e42b1e235f5e050a265 WatchSource:0}: Error finding container c185d891fff98e841d23289b330b89ec62882f4a058a6e42b1e235f5e050a265: Status 404 returned error can't find the container with id c185d891fff98e841d23289b330b89ec62882f4a058a6e42b1e235f5e050a265 Nov 01 06:27:28.528685 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:28.528654 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk"] Nov 01 06:27:28.530971 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:28.530935 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:27:28.531109 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:28.531016 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:27:28.531109 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:28.531060 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:27:29.024468 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:29.024433 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" event={"ID":"f2e934b9-b834-487b-9a7b-08f3047c49c6","Type":"ContainerStarted","Data":"2dd9ae47500e93ec5adb451632795064e3ce242c61f8461f6f04a6931751c19d"} Nov 01 06:27:29.024968 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:29.024525 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" event={"ID":"f2e934b9-b834-487b-9a7b-08f3047c49c6","Type":"ContainerStarted","Data":"c185d891fff98e841d23289b330b89ec62882f4a058a6e42b1e235f5e050a265"} Nov 01 06:27:29.090660 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:29.090618 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:27:29.097261 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:29.097224 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:27:29.148999 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:29.148942 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" podStartSLOduration=2.14892687 podStartE2EDuration="2.14892687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:27:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:27:29.14685353 +0000 UTC m=+2527.801970698" watchObservedRunningTime="2026-11-01 06:27:29.14892687 +0000 UTC m=+2527.804044036" Nov 01 06:27:30.031331 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:30.031288 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:27:30.032220 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:27:30.032199 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:27:39.973628 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:27:39.973580 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:28:03.390168 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:03.390120 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" podUID="75afe340-0b9c-4b4c-89ab-20d761e6b4f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:28:03.425435 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:03.423593 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" podUID="1878b6fb-f8fe-4f4f-8f4e-056c66fc206a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:28:03.480692 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:03.480625 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc" podUID="66fdf9da-4790-40c5-9434-052c7465d23e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:28:03.503596 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:03.502728 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" podUID="adbac784-4ded-4ca3-b985-3e59983fc637" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 01 06:28:03.523127 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:03.523067 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-brtbn" podUID="f94f4767-3f84-429a-bff5-38c4fa82deaa" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.41:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 06:28:03.586526 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:03.586025 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" podUID="57cf5e07-36de-48cf-b938-138484cea3a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:28:04.246971 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:04.246592 5040 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" podUID="5daa03ca-824b-4799-a11b-4b3af1512c38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.123:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 06:28:09.979767 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:28:09.979728 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:28:11.503825 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:11.503789 5040 generic.go:356] "Generic (PLEG): container finished" podID="61a726a8-a06e-4f27-95e9-bca8b6b78f6d" containerID="c91c6d21f50b1bd24936f61867715b6115e4459b30f898aba28f2db276c632ef" exitCode=0 Nov 01 06:28:11.504351 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:11.503840 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" event={"ID":"61a726a8-a06e-4f27-95e9-bca8b6b78f6d","Type":"ContainerDied","Data":"c91c6d21f50b1bd24936f61867715b6115e4459b30f898aba28f2db276c632ef"} Nov 01 06:28:11.504351 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:11.504280 5040 scope.go:122] "RemoveContainer" containerID="c91c6d21f50b1bd24936f61867715b6115e4459b30f898aba28f2db276c632ef" Nov 01 06:28:11.505821 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:11.505791 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:28:11.505920 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:11.505860 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:28:11.505920 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:11.505894 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:28:12.526280 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:12.526242 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" event={"ID":"61a726a8-a06e-4f27-95e9-bca8b6b78f6d","Type":"ContainerStarted","Data":"d74e94faf585a6511b825684b07542bbdf47baab2f4f0bc80a198fc2c9d97709"} Nov 01 06:28:13.521090 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:13.521047 5040 patch_prober.go:29] interesting pod/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 01 06:28:13.521346 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:13.521126 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" podUID="61a726a8-a06e-4f27-95e9-bca8b6b78f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 01 06:28:13.526125 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:13.526097 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:28:14.519020 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:14.518973 5040 patch_prober.go:29] interesting pod/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 01 06:28:14.519544 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:14.519057 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" podUID="61a726a8-a06e-4f27-95e9-bca8b6b78f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 01 06:28:15.504981 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:15.504937 5040 patch_prober.go:29] interesting pod/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 01 06:28:15.506117 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:15.506083 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" podUID="61a726a8-a06e-4f27-95e9-bca8b6b78f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 01 06:28:16.512504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:16.507946 5040 patch_prober.go:29] interesting pod/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 01 06:28:16.512504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:16.508028 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" podUID="61a726a8-a06e-4f27-95e9-bca8b6b78f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 01 06:28:17.513506 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:17.511356 5040 patch_prober.go:29] interesting pod/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 01 06:28:17.513506 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:17.511458 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" podUID="61a726a8-a06e-4f27-95e9-bca8b6b78f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 01 06:28:18.508267 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:18.508225 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:28:18.508743 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:18.508715 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:28:18.509377 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:18.509359 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:28:29.260758 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:29.260717 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp"] Nov 01 06:28:29.264366 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:29.264347 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:28:29.319627 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:29.319595 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-d7d444-openshift-default-dockercfg-lc7ft\"" Nov 01 06:28:29.375795 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:29.375754 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp"] Nov 01 06:28:29.409572 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:29.409532 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb482d61-0774-4394-8c37-a564ef5b6838-credential-socket\") pod \"e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp\" (UID: \"bb482d61-0774-4394-8c37-a564ef5b6838\") " pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:28:29.417726 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:29.417687 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2pwq\" (UniqueName: \"kubernetes.io/projected/bb482d61-0774-4394-8c37-a564ef5b6838-kube-api-access-k2pwq\") pod \"e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp\" (UID: \"bb482d61-0774-4394-8c37-a564ef5b6838\") " pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:28:29.417953 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:29.417743 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb482d61-0774-4394-8c37-a564ef5b6838-workload-certs\") pod \"e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp\" (UID: \"bb482d61-0774-4394-8c37-a564ef5b6838\") " pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:28:29.417953 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:29.417797 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb482d61-0774-4394-8c37-a564ef5b6838-workload-socket\") pod \"e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp\" (UID: \"bb482d61-0774-4394-8c37-a564ef5b6838\") " pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:28:29.417953 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:29.417813 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb482d61-0774-4394-8c37-a564ef5b6838-istio-data\") pod \"e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp\" (UID: \"bb482d61-0774-4394-8c37-a564ef5b6838\") " pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:28:29.417953 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:29.417847 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb482d61-0774-4394-8c37-a564ef5b6838-istiod-ca-cert\") pod \"e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp\" (UID: \"bb482d61-0774-4394-8c37-a564ef5b6838\") " pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:28:29.417953 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:29.417904 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb482d61-0774-4394-8c37-a564ef5b6838-istio-token\") pod \"e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp\" (UID: \"bb482d61-0774-4394-8c37-a564ef5b6838\") " pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:28:29.417953 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:29.417944 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb482d61-0774-4394-8c37-a564ef5b6838-istio-podinfo\") pod \"e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp\" (UID: \"bb482d61-0774-4394-8c37-a564ef5b6838\") " pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:28:29.418175 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:29.417982 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb482d61-0774-4394-8c37-a564ef5b6838-istio-envoy\") pod \"e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp\" (UID: \"bb482d61-0774-4394-8c37-a564ef5b6838\") " pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:28:29.519274 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:29.519166 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2pwq\" (UniqueName: \"kubernetes.io/projected/bb482d61-0774-4394-8c37-a564ef5b6838-kube-api-access-k2pwq\") pod \"e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp\" (UID: \"bb482d61-0774-4394-8c37-a564ef5b6838\") " pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:28:29.519274 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:29.519232 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb482d61-0774-4394-8c37-a564ef5b6838-workload-certs\") pod \"e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp\" (UID: \"bb482d61-0774-4394-8c37-a564ef5b6838\") " pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:28:29.519547 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:29.519298 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb482d61-0774-4394-8c37-a564ef5b6838-workload-socket\") pod \"e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp\" (UID: \"bb482d61-0774-4394-8c37-a564ef5b6838\") " pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:28:29.519547 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:29.519323 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb482d61-0774-4394-8c37-a564ef5b6838-istio-data\") pod \"e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp\" (UID: \"bb482d61-0774-4394-8c37-a564ef5b6838\") " pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:28:29.519547 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:29.519368 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb482d61-0774-4394-8c37-a564ef5b6838-istiod-ca-cert\") pod \"e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp\" (UID: \"bb482d61-0774-4394-8c37-a564ef5b6838\") " pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:28:29.519547 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:29.519393 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb482d61-0774-4394-8c37-a564ef5b6838-istio-token\") pod \"e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp\" (UID: \"bb482d61-0774-4394-8c37-a564ef5b6838\") " pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:28:29.519547 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:29.519441 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb482d61-0774-4394-8c37-a564ef5b6838-istio-podinfo\") pod \"e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp\" (UID: \"bb482d61-0774-4394-8c37-a564ef5b6838\") " pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:28:29.519547 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:29.519511 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb482d61-0774-4394-8c37-a564ef5b6838-istio-envoy\") pod \"e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp\" (UID: \"bb482d61-0774-4394-8c37-a564ef5b6838\") " pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:28:29.519892 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:29.519559 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb482d61-0774-4394-8c37-a564ef5b6838-credential-socket\") pod \"e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp\" (UID: \"bb482d61-0774-4394-8c37-a564ef5b6838\") " pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:28:29.520609 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:29.520570 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb482d61-0774-4394-8c37-a564ef5b6838-credential-socket\") pod \"e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp\" (UID: \"bb482d61-0774-4394-8c37-a564ef5b6838\") " pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:28:29.521267 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:29.521238 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb482d61-0774-4394-8c37-a564ef5b6838-workload-certs\") pod \"e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp\" (UID: \"bb482d61-0774-4394-8c37-a564ef5b6838\") " pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:28:29.521536 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:29.521511 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb482d61-0774-4394-8c37-a564ef5b6838-workload-socket\") pod \"e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp\" (UID: \"bb482d61-0774-4394-8c37-a564ef5b6838\") " pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:28:29.521788 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:29.521750 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb482d61-0774-4394-8c37-a564ef5b6838-istio-data\") pod \"e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp\" (UID: \"bb482d61-0774-4394-8c37-a564ef5b6838\") " pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:28:29.523746 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:29.523711 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb482d61-0774-4394-8c37-a564ef5b6838-istiod-ca-cert\") pod \"e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp\" (UID: \"bb482d61-0774-4394-8c37-a564ef5b6838\") " pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:28:29.526576 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:29.526550 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb482d61-0774-4394-8c37-a564ef5b6838-istio-envoy\") pod \"e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp\" (UID: \"bb482d61-0774-4394-8c37-a564ef5b6838\") " pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:28:29.526754 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:29.526729 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb482d61-0774-4394-8c37-a564ef5b6838-istio-podinfo\") pod \"e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp\" (UID: \"bb482d61-0774-4394-8c37-a564ef5b6838\") " pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:28:29.582573 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:29.582541 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2pwq\" (UniqueName: \"kubernetes.io/projected/bb482d61-0774-4394-8c37-a564ef5b6838-kube-api-access-k2pwq\") pod \"e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp\" (UID: \"bb482d61-0774-4394-8c37-a564ef5b6838\") " pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:28:29.584272 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:29.584249 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb482d61-0774-4394-8c37-a564ef5b6838-istio-token\") pod \"e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp\" (UID: \"bb482d61-0774-4394-8c37-a564ef5b6838\") " pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:28:29.877648 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:29.877556 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:28:30.308145 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:30.308110 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp"] Nov 01 06:28:30.309032 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:28:30.309003 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb482d61_0774_4394_8c37_a564ef5b6838.slice/crio-12dc377a1d60b60ce112296d4125edfea9d93846a1432702b4fbcc082927cb34 WatchSource:0}: Error finding container 12dc377a1d60b60ce112296d4125edfea9d93846a1432702b4fbcc082927cb34: Status 404 returned error can't find the container with id 12dc377a1d60b60ce112296d4125edfea9d93846a1432702b4fbcc082927cb34 Nov 01 06:28:30.311626 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:30.311592 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:28:30.311719 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:30.311664 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:28:30.311719 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:30.311694 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:28:30.661771 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:30.661673 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" event={"ID":"bb482d61-0774-4394-8c37-a564ef5b6838","Type":"ContainerStarted","Data":"a83bdbd5fe62ee118d2b1f3f325489a615d4b59532599da5a32e0f396fd47809"} Nov 01 06:28:30.661771 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:30.661742 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" event={"ID":"bb482d61-0774-4394-8c37-a564ef5b6838","Type":"ContainerStarted","Data":"12dc377a1d60b60ce112296d4125edfea9d93846a1432702b4fbcc082927cb34"} Nov 01 06:28:30.785641 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:30.785585 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" podStartSLOduration=1.785571899 podStartE2EDuration="1.785571899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:28:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:28:30.782671335 +0000 UTC m=+2589.437788503" watchObservedRunningTime="2026-11-01 06:28:30.785571899 +0000 UTC m=+2589.440689060" Nov 01 06:28:30.878321 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:30.878283 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:28:31.884538 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:31.884507 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:28:32.675135 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:32.675099 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:28:32.676287 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:32.676263 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:28:34.447504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:34.446576 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-brrdl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 06:28:34.447504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:34.446643 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" podUID="b0ce9454-b676-4e7d-abd3-34d76aa2c787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 06:28:38.717969 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:38.717928 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce346503-48g2v"] Nov 01 06:28:38.721948 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:38.721922 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce346503-48g2v" Nov 01 06:28:38.780612 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:38.780568 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce346503-48g2v"] Nov 01 06:28:38.820988 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:38.820946 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdt8z\" (UniqueName: \"kubernetes.io/projected/7ab071fa-abe3-431d-930e-f69b97d59e69-kube-api-access-tdt8z\") pod \"maas-api-revoke-keys-e2e-ait-ce346503-48g2v\" (UID: \"7ab071fa-abe3-431d-930e-f69b97d59e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce346503-48g2v" Nov 01 06:28:38.821166 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:38.821092 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ab071fa-abe3-431d-930e-f69b97d59e69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ce346503-48g2v\" (UID: \"7ab071fa-abe3-431d-930e-f69b97d59e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce346503-48g2v" Nov 01 06:28:38.821239 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:38.821215 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ab071fa-abe3-431d-930e-f69b97d59e69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ce346503-48g2v\" (UID: \"7ab071fa-abe3-431d-930e-f69b97d59e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce346503-48g2v" Nov 01 06:28:38.825341 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:38.825317 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:28:38.825597 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:38.825580 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:28:38.825661 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:38.825620 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:28:38.825661 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:38.825646 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:28:38.921871 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:38.921836 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ab071fa-abe3-431d-930e-f69b97d59e69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ce346503-48g2v\" (UID: \"7ab071fa-abe3-431d-930e-f69b97d59e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce346503-48g2v" Nov 01 06:28:38.922051 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:38.921914 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ab071fa-abe3-431d-930e-f69b97d59e69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ce346503-48g2v\" (UID: \"7ab071fa-abe3-431d-930e-f69b97d59e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce346503-48g2v" Nov 01 06:28:38.922051 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:38.921971 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdt8z\" (UniqueName: \"kubernetes.io/projected/7ab071fa-abe3-431d-930e-f69b97d59e69-kube-api-access-tdt8z\") pod \"maas-api-revoke-keys-e2e-ait-ce346503-48g2v\" (UID: \"7ab071fa-abe3-431d-930e-f69b97d59e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce346503-48g2v" Nov 01 06:28:38.922660 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:38.922632 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ab071fa-abe3-431d-930e-f69b97d59e69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ce346503-48g2v\" (UID: \"7ab071fa-abe3-431d-930e-f69b97d59e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce346503-48g2v" Nov 01 06:28:38.924505 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:38.924466 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ab071fa-abe3-431d-930e-f69b97d59e69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ce346503-48g2v\" (UID: \"7ab071fa-abe3-431d-930e-f69b97d59e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce346503-48g2v" Nov 01 06:28:38.984105 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:38.984030 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdt8z\" (UniqueName: \"kubernetes.io/projected/7ab071fa-abe3-431d-930e-f69b97d59e69-kube-api-access-tdt8z\") pod \"maas-api-revoke-keys-e2e-ait-ce346503-48g2v\" (UID: \"7ab071fa-abe3-431d-930e-f69b97d59e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce346503-48g2v" Nov 01 06:28:39.035678 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:39.035639 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce346503-48g2v" Nov 01 06:28:39.541668 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:39.541631 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce346503-48g2v"] Nov 01 06:28:39.764986 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:39.764902 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce346503-48g2v" event={"ID":"7ab071fa-abe3-431d-930e-f69b97d59e69","Type":"ContainerStarted","Data":"c822295707f31422f7431ba462b5820600838b72efa2a5ed5e5bc61b29b720a4"} Nov 01 06:28:39.973842 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:28:39.973805 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:28:40.781008 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:40.780955 5040 generic.go:356] "Generic (PLEG): container finished" podID="7ab071fa-abe3-431d-930e-f69b97d59e69" containerID="6c22ce6cbbe3f48321a5db1cc3e917eb270257959be51bafc1ddc3890b875202" exitCode=0 Nov 01 06:28:40.781522 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:40.781090 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce346503-48g2v" event={"ID":"7ab071fa-abe3-431d-930e-f69b97d59e69","Type":"ContainerDied","Data":"6c22ce6cbbe3f48321a5db1cc3e917eb270257959be51bafc1ddc3890b875202"} Nov 01 06:28:41.944878 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:41.944851 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce346503-48g2v" Nov 01 06:28:42.059359 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:42.059266 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ab071fa-abe3-431d-930e-f69b97d59e69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ab071fa-abe3-431d-930e-f69b97d59e69-maas-api-openshift-ca\") pod \"7ab071fa-abe3-431d-930e-f69b97d59e69\" (UID: \"7ab071fa-abe3-431d-930e-f69b97d59e69\") " Nov 01 06:28:42.059359 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:42.059350 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ab071fa-abe3-431d-930e-f69b97d59e69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ab071fa-abe3-431d-930e-f69b97d59e69-maas-api-xks-ca\") pod \"7ab071fa-abe3-431d-930e-f69b97d59e69\" (UID: \"7ab071fa-abe3-431d-930e-f69b97d59e69\") " Nov 01 06:28:42.059668 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:42.059385 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ab071fa-abe3-431d-930e-f69b97d59e69-kube-api-access-tdt8z\" (UniqueName: \"kubernetes.io/projected/7ab071fa-abe3-431d-930e-f69b97d59e69-kube-api-access-tdt8z\") pod \"7ab071fa-abe3-431d-930e-f69b97d59e69\" (UID: \"7ab071fa-abe3-431d-930e-f69b97d59e69\") " Nov 01 06:28:42.059740 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:42.059716 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ab071fa-abe3-431d-930e-f69b97d59e69-maas-api-openshift-ca" pod "7ab071fa-abe3-431d-930e-f69b97d59e69" (UID: "7ab071fa-abe3-431d-930e-f69b97d59e69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:28:42.062064 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:42.062033 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ab071fa-abe3-431d-930e-f69b97d59e69-kube-api-access-tdt8z" pod "7ab071fa-abe3-431d-930e-f69b97d59e69" (UID: "7ab071fa-abe3-431d-930e-f69b97d59e69"). InnerVolumeSpecName "kube-api-access-tdt8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:28:42.065998 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:42.065967 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ab071fa-abe3-431d-930e-f69b97d59e69-maas-api-xks-ca" pod "7ab071fa-abe3-431d-930e-f69b97d59e69" (UID: "7ab071fa-abe3-431d-930e-f69b97d59e69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:28:42.160592 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:42.160550 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ab071fa-abe3-431d-930e-f69b97d59e69-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:28:42.160592 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:42.160597 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdt8z\" (UniqueName: \"kubernetes.io/projected/7ab071fa-abe3-431d-930e-f69b97d59e69-kube-api-access-tdt8z\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:28:42.161044 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:42.160609 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ab071fa-abe3-431d-930e-f69b97d59e69-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:28:42.795905 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:42.795869 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce346503-48g2v" Nov 01 06:28:42.796097 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:42.795905 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce346503-48g2v" event={"ID":"7ab071fa-abe3-431d-930e-f69b97d59e69","Type":"ContainerDied","Data":"c822295707f31422f7431ba462b5820600838b72efa2a5ed5e5bc61b29b720a4"} Nov 01 06:28:42.796097 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:42.795960 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c822295707f31422f7431ba462b5820600838b72efa2a5ed5e5bc61b29b720a4" Nov 01 06:28:43.357737 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:43.357695 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz"] Nov 01 06:28:43.358410 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:43.358386 5040 cpu_manager.go:414] "containerMap: removing container" podUID="7ab071fa-abe3-431d-930e-f69b97d59e69" containerName="revoke-keys" Nov 01 06:28:43.358524 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:43.358414 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ab071fa-abe3-431d-930e-f69b97d59e69" containerName="revoke-keys" Nov 01 06:28:43.358638 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:43.358622 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ab071fa-abe3-431d-930e-f69b97d59e69" containerName="revoke-keys" Nov 01 06:28:43.362529 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:43.362470 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:28:43.449032 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:43.449004 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c0662f25-openshift-default-dockercfg-lff96\"" Nov 01 06:28:43.479737 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:43.479698 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd8zz\" (UniqueName: \"kubernetes.io/projected/1978f64d-b811-47bf-862a-aacbde03b6e9-kube-api-access-bd8zz\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-tskhz\" (UID: \"1978f64d-b811-47bf-862a-aacbde03b6e9\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:28:43.479941 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:43.479822 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1978f64d-b811-47bf-862a-aacbde03b6e9-workload-certs\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-tskhz\" (UID: \"1978f64d-b811-47bf-862a-aacbde03b6e9\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:28:43.480030 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:43.479971 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1978f64d-b811-47bf-862a-aacbde03b6e9-istio-token\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-tskhz\" (UID: \"1978f64d-b811-47bf-862a-aacbde03b6e9\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:28:43.480110 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:43.480090 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1978f64d-b811-47bf-862a-aacbde03b6e9-workload-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-tskhz\" (UID: \"1978f64d-b811-47bf-862a-aacbde03b6e9\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:28:43.480175 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:43.480149 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1978f64d-b811-47bf-862a-aacbde03b6e9-istio-podinfo\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-tskhz\" (UID: \"1978f64d-b811-47bf-862a-aacbde03b6e9\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:28:43.480246 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:43.480211 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1978f64d-b811-47bf-862a-aacbde03b6e9-istio-envoy\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-tskhz\" (UID: \"1978f64d-b811-47bf-862a-aacbde03b6e9\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:28:43.480312 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:43.480274 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1978f64d-b811-47bf-862a-aacbde03b6e9-istio-data\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-tskhz\" (UID: \"1978f64d-b811-47bf-862a-aacbde03b6e9\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:28:43.480398 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:43.480376 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1978f64d-b811-47bf-862a-aacbde03b6e9-istiod-ca-cert\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-tskhz\" (UID: \"1978f64d-b811-47bf-862a-aacbde03b6e9\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:28:43.480494 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:43.480425 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1978f64d-b811-47bf-862a-aacbde03b6e9-credential-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-tskhz\" (UID: \"1978f64d-b811-47bf-862a-aacbde03b6e9\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:28:43.506372 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:43.506326 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz"] Nov 01 06:28:43.589669 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:43.588120 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1978f64d-b811-47bf-862a-aacbde03b6e9-istio-podinfo\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-tskhz\" (UID: \"1978f64d-b811-47bf-862a-aacbde03b6e9\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:28:43.589669 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:43.588191 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1978f64d-b811-47bf-862a-aacbde03b6e9-istio-envoy\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-tskhz\" (UID: \"1978f64d-b811-47bf-862a-aacbde03b6e9\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:28:43.589669 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:43.588216 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1978f64d-b811-47bf-862a-aacbde03b6e9-istio-data\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-tskhz\" (UID: \"1978f64d-b811-47bf-862a-aacbde03b6e9\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:28:43.589669 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:43.588293 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1978f64d-b811-47bf-862a-aacbde03b6e9-istiod-ca-cert\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-tskhz\" (UID: \"1978f64d-b811-47bf-862a-aacbde03b6e9\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:28:43.589669 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:43.588339 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1978f64d-b811-47bf-862a-aacbde03b6e9-credential-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-tskhz\" (UID: \"1978f64d-b811-47bf-862a-aacbde03b6e9\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:28:43.589669 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:43.588417 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd8zz\" (UniqueName: \"kubernetes.io/projected/1978f64d-b811-47bf-862a-aacbde03b6e9-kube-api-access-bd8zz\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-tskhz\" (UID: \"1978f64d-b811-47bf-862a-aacbde03b6e9\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:28:43.589669 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:43.588449 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1978f64d-b811-47bf-862a-aacbde03b6e9-workload-certs\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-tskhz\" (UID: \"1978f64d-b811-47bf-862a-aacbde03b6e9\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:28:43.589669 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:43.588543 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1978f64d-b811-47bf-862a-aacbde03b6e9-istio-token\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-tskhz\" (UID: \"1978f64d-b811-47bf-862a-aacbde03b6e9\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:28:43.589669 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:43.588617 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1978f64d-b811-47bf-862a-aacbde03b6e9-workload-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-tskhz\" (UID: \"1978f64d-b811-47bf-862a-aacbde03b6e9\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:28:43.589669 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:43.589116 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1978f64d-b811-47bf-862a-aacbde03b6e9-workload-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-tskhz\" (UID: \"1978f64d-b811-47bf-862a-aacbde03b6e9\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:28:43.591035 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:43.591004 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1978f64d-b811-47bf-862a-aacbde03b6e9-credential-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-tskhz\" (UID: \"1978f64d-b811-47bf-862a-aacbde03b6e9\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:28:43.591873 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:43.591851 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1978f64d-b811-47bf-862a-aacbde03b6e9-workload-certs\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-tskhz\" (UID: \"1978f64d-b811-47bf-862a-aacbde03b6e9\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:28:43.597232 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:43.593605 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1978f64d-b811-47bf-862a-aacbde03b6e9-istiod-ca-cert\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-tskhz\" (UID: \"1978f64d-b811-47bf-862a-aacbde03b6e9\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:28:43.597411 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:43.597071 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1978f64d-b811-47bf-862a-aacbde03b6e9-istio-envoy\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-tskhz\" (UID: \"1978f64d-b811-47bf-862a-aacbde03b6e9\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:28:43.597618 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:43.597145 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1978f64d-b811-47bf-862a-aacbde03b6e9-istio-data\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-tskhz\" (UID: \"1978f64d-b811-47bf-862a-aacbde03b6e9\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:28:43.598877 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:43.598855 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1978f64d-b811-47bf-862a-aacbde03b6e9-istio-podinfo\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-tskhz\" (UID: \"1978f64d-b811-47bf-862a-aacbde03b6e9\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:28:43.654226 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:43.654144 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1978f64d-b811-47bf-862a-aacbde03b6e9-istio-token\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-tskhz\" (UID: \"1978f64d-b811-47bf-862a-aacbde03b6e9\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:28:43.654753 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:43.654729 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd8zz\" (UniqueName: \"kubernetes.io/projected/1978f64d-b811-47bf-862a-aacbde03b6e9-kube-api-access-bd8zz\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-tskhz\" (UID: \"1978f64d-b811-47bf-862a-aacbde03b6e9\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:28:43.688831 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:43.688777 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:28:44.136519 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:28:44.136470 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1978f64d_b811_47bf_862a_aacbde03b6e9.slice/crio-c4411af092eccd7b4265f3e4f7f8cb6890c0898681e8f4b4b1fe6b9ba62f247b WatchSource:0}: Error finding container c4411af092eccd7b4265f3e4f7f8cb6890c0898681e8f4b4b1fe6b9ba62f247b: Status 404 returned error can't find the container with id c4411af092eccd7b4265f3e4f7f8cb6890c0898681e8f4b4b1fe6b9ba62f247b Nov 01 06:28:44.136681 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:44.136660 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz"] Nov 01 06:28:44.139032 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:44.139000 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:28:44.139112 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:44.139063 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:28:44.139112 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:44.139094 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:28:44.814882 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:44.814841 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" event={"ID":"1978f64d-b811-47bf-862a-aacbde03b6e9","Type":"ContainerStarted","Data":"d007250bc3884dd71ace59931222b603f24b26e9f439455797992cd76c981cc4"} Nov 01 06:28:44.815333 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:44.814905 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" event={"ID":"1978f64d-b811-47bf-862a-aacbde03b6e9","Type":"ContainerStarted","Data":"c4411af092eccd7b4265f3e4f7f8cb6890c0898681e8f4b4b1fe6b9ba62f247b"} Nov 01 06:28:45.046254 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:45.044181 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" podStartSLOduration=2.044159177 podStartE2EDuration="2.044159177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:28:44.986431158 +0000 UTC m=+2603.641548326" watchObservedRunningTime="2026-11-01 06:28:45.044159177 +0000 UTC m=+2603.699276345" Nov 01 06:28:45.689231 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:45.689188 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:28:45.695546 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:45.695517 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:28:45.825100 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:45.825068 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:28:45.826118 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:45.826090 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:28:55.904374 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:55.904340 5040 generic.go:356] "Generic (PLEG): container finished" podID="f2e934b9-b834-487b-9a7b-08f3047c49c6" containerID="2dd9ae47500e93ec5adb451632795064e3ce242c61f8461f6f04a6931751c19d" exitCode=0 Nov 01 06:28:55.904850 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:55.904409 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" event={"ID":"f2e934b9-b834-487b-9a7b-08f3047c49c6","Type":"ContainerDied","Data":"2dd9ae47500e93ec5adb451632795064e3ce242c61f8461f6f04a6931751c19d"} Nov 01 06:28:55.904850 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:55.904829 5040 scope.go:122] "RemoveContainer" containerID="2dd9ae47500e93ec5adb451632795064e3ce242c61f8461f6f04a6931751c19d" Nov 01 06:28:55.906419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:55.906391 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:28:55.906544 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:55.906456 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:28:55.906544 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:55.906502 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:28:56.925549 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:56.925506 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" event={"ID":"f2e934b9-b834-487b-9a7b-08f3047c49c6","Type":"ContainerStarted","Data":"0b0140b0931a87b10ae9297a6e3f35854cbe6c0a697e52f9a8d4c8e9d9249909"} Nov 01 06:28:57.090636 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:57.090593 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:28:57.091984 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:57.091957 5040 patch_prober.go:29] interesting pod/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 01 06:28:57.092106 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:57.092033 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" podUID="f2e934b9-b834-487b-9a7b-08f3047c49c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 01 06:28:58.090862 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:58.090825 5040 patch_prober.go:29] interesting pod/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 01 06:28:58.091260 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:58.090877 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" podUID="f2e934b9-b834-487b-9a7b-08f3047c49c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 01 06:28:59.091223 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:59.091183 5040 patch_prober.go:29] interesting pod/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 01 06:28:59.091687 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:28:59.091240 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" podUID="f2e934b9-b834-487b-9a7b-08f3047c49c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 01 06:29:00.031960 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:00.031918 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:29:00.090885 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:00.090844 5040 patch_prober.go:29] interesting pod/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 01 06:29:00.091106 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:00.090910 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" podUID="f2e934b9-b834-487b-9a7b-08f3047c49c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 01 06:29:01.090847 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:01.090801 5040 patch_prober.go:29] interesting pod/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 01 06:29:01.091292 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:01.090874 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" podUID="f2e934b9-b834-487b-9a7b-08f3047c49c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 01 06:29:02.097429 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:02.097388 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:29:02.098888 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:02.098865 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:29:04.049546 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:04.049511 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c69393ab-kl5js"] Nov 01 06:29:04.100177 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:04.100138 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c69393ab-kl5js"] Nov 01 06:29:05.515458 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:05.515420 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dc14a58-dd7a-4024-860b-ad7de008bc36" path="/var/lib/kubelet/pods/3dc14a58-dd7a-4024-860b-ad7de008bc36/volumes" Nov 01 06:29:07.048453 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:07.048415 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40f0d38f-dqxbg"] Nov 01 06:29:07.059076 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:07.059042 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40f0d38f-dqxbg"] Nov 01 06:29:07.515279 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:07.515241 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c3629eb-bb73-46e3-b7a9-9447a92cd1c7" path="/var/lib/kubelet/pods/6c3629eb-bb73-46e3-b7a9-9447a92cd1c7/volumes" Nov 01 06:29:09.973251 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:29:09.973204 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:29:17.050874 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:17.050828 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-489da514-7c9d7"] Nov 01 06:29:17.101519 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:17.101469 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-489da514-7c9d7"] Nov 01 06:29:17.515057 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:17.515019 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb003776-c6cd-4f28-966e-9c3b91dd19cd" path="/var/lib/kubelet/pods/bb003776-c6cd-4f28-966e-9c3b91dd19cd/volumes" Nov 01 06:29:18.049976 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:18.049934 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdfb2df7-qvfqw"] Nov 01 06:29:18.100041 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:18.099999 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdfb2df7-qvfqw"] Nov 01 06:29:19.515113 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:19.515082 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41406be2-24cd-449c-90dc-a56a1d07cb90" path="/var/lib/kubelet/pods/41406be2-24cd-449c-90dc-a56a1d07cb90/volumes" Nov 01 06:29:22.951684 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:22.951651 5040 scope.go:122] "RemoveContainer" containerID="943f772bb1d77d54cbd0730408d82de75a8ef3b63a9eaaf23221a9e01495fc9f" Nov 01 06:29:32.194529 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:32.194417 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce346503-48g2v"] Nov 01 06:29:32.244770 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:32.244735 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce346503-48g2v"] Nov 01 06:29:32.779331 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:32.779291 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv"] Nov 01 06:29:32.779536 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:32.779499 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" podUID="61a726a8-a06e-4f27-95e9-bca8b6b78f6d" containerName="istio-proxy" containerID="cri-o://d74e94faf585a6511b825684b07542bbdf47baab2f4f0bc80a198fc2c9d97709" gracePeriod=30 Nov 01 06:29:33.511340 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:33.511287 5040 patch_prober.go:29] interesting pod/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 06:29:33.511823 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:33.511361 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" podUID="61a726a8-a06e-4f27-95e9-bca8b6b78f6d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 06:29:33.517880 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:33.517850 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ab071fa-abe3-431d-930e-f69b97d59e69" path="/var/lib/kubelet/pods/7ab071fa-abe3-431d-930e-f69b97d59e69/volumes" Nov 01 06:29:38.048309 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.048225 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5507e36a-c2jzw"] Nov 01 06:29:38.055729 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.055705 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:29:38.097077 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.097042 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-istio-envoy\") pod \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\" (UID: \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\") " Nov 01 06:29:38.097257 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.097093 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-workload-certs\") pod \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\" (UID: \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\") " Nov 01 06:29:38.097257 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.097146 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-istio-token\" (UniqueName: \"kubernetes.io/projected/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-istio-token\") pod \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\" (UID: \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\") " Nov 01 06:29:38.097257 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.097202 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-istio-data\") pod \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\" (UID: \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\") " Nov 01 06:29:38.097257 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.097223 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-istio-podinfo\") pod \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\" (UID: \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\") " Nov 01 06:29:38.097257 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.097250 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-credential-socket\") pod \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\" (UID: \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\") " Nov 01 06:29:38.097567 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.097284 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-istiod-ca-cert\") pod \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\" (UID: \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\") " Nov 01 06:29:38.097567 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.097378 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-kube-api-access-c2cwx\" (UniqueName: \"kubernetes.io/projected/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-kube-api-access-c2cwx\") pod \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\" (UID: \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\") " Nov 01 06:29:38.097567 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.097398 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-workload-certs" pod "61a726a8-a06e-4f27-95e9-bca8b6b78f6d" (UID: "61a726a8-a06e-4f27-95e9-bca8b6b78f6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:29:38.097567 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.097430 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-workload-socket\") pod \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\" (UID: \"61a726a8-a06e-4f27-95e9-bca8b6b78f6d\") " Nov 01 06:29:38.097793 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.097616 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-istio-data" pod "61a726a8-a06e-4f27-95e9-bca8b6b78f6d" (UID: "61a726a8-a06e-4f27-95e9-bca8b6b78f6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:29:38.097849 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.097807 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-workload-socket" pod "61a726a8-a06e-4f27-95e9-bca8b6b78f6d" (UID: "61a726a8-a06e-4f27-95e9-bca8b6b78f6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:29:38.097904 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.097848 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-istiod-ca-cert" pod "61a726a8-a06e-4f27-95e9-bca8b6b78f6d" (UID: "61a726a8-a06e-4f27-95e9-bca8b6b78f6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:29:38.098072 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.098049 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-credential-socket" pod "61a726a8-a06e-4f27-95e9-bca8b6b78f6d" (UID: "61a726a8-a06e-4f27-95e9-bca8b6b78f6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:29:38.098141 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.098058 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:29:38.098141 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.098096 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:29:38.098141 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.098123 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:29:38.098141 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.098138 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:29:38.098838 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.098814 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5507e36a-c2jzw"] Nov 01 06:29:38.099447 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.099416 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-istio-envoy" pod "61a726a8-a06e-4f27-95e9-bca8b6b78f6d" (UID: "61a726a8-a06e-4f27-95e9-bca8b6b78f6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:29:38.099709 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.099691 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-kube-api-access-c2cwx" pod "61a726a8-a06e-4f27-95e9-bca8b6b78f6d" (UID: "61a726a8-a06e-4f27-95e9-bca8b6b78f6d"). InnerVolumeSpecName "kube-api-access-c2cwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:29:38.099838 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.099820 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-istio-podinfo" pod "61a726a8-a06e-4f27-95e9-bca8b6b78f6d" (UID: "61a726a8-a06e-4f27-95e9-bca8b6b78f6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:29:38.099881 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.099835 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-istio-token" pod "61a726a8-a06e-4f27-95e9-bca8b6b78f6d" (UID: "61a726a8-a06e-4f27-95e9-bca8b6b78f6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:29:38.199528 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.199467 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2cwx\" (UniqueName: \"kubernetes.io/projected/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-kube-api-access-c2cwx\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:29:38.199528 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.199521 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:29:38.199528 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.199531 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:29:38.199832 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.199540 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:29:38.199832 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.199549 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a726a8-a06e-4f27-95e9-bca8b6b78f6d-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:29:38.239776 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.239743 5040 generic.go:356] "Generic (PLEG): container finished" podID="61a726a8-a06e-4f27-95e9-bca8b6b78f6d" containerID="d74e94faf585a6511b825684b07542bbdf47baab2f4f0bc80a198fc2c9d97709" exitCode=0 Nov 01 06:29:38.239941 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.239796 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" Nov 01 06:29:38.239941 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.239817 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" event={"ID":"61a726a8-a06e-4f27-95e9-bca8b6b78f6d","Type":"ContainerDied","Data":"d74e94faf585a6511b825684b07542bbdf47baab2f4f0bc80a198fc2c9d97709"} Nov 01 06:29:38.239941 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.239885 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv" event={"ID":"61a726a8-a06e-4f27-95e9-bca8b6b78f6d","Type":"ContainerDied","Data":"3df9ed674a866a1ceaadefca0d243d87a46e70a12d5bf2c02a0d8d91e59c22dd"} Nov 01 06:29:38.239941 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.239909 5040 scope.go:122] "RemoveContainer" containerID="d74e94faf585a6511b825684b07542bbdf47baab2f4f0bc80a198fc2c9d97709" Nov 01 06:29:38.250931 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.250915 5040 scope.go:122] "RemoveContainer" containerID="c91c6d21f50b1bd24936f61867715b6115e4459b30f898aba28f2db276c632ef" Nov 01 06:29:38.261146 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.261111 5040 scope.go:122] "RemoveContainer" containerID="d74e94faf585a6511b825684b07542bbdf47baab2f4f0bc80a198fc2c9d97709" Nov 01 06:29:38.261467 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:29:38.261444 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d74e94faf585a6511b825684b07542bbdf47baab2f4f0bc80a198fc2c9d97709\": container with ID starting with d74e94faf585a6511b825684b07542bbdf47baab2f4f0bc80a198fc2c9d97709 not found: ID does not exist" containerID="d74e94faf585a6511b825684b07542bbdf47baab2f4f0bc80a198fc2c9d97709" Nov 01 06:29:38.261580 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.261469 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d74e94faf585a6511b825684b07542bbdf47baab2f4f0bc80a198fc2c9d97709"} err="failed to get container status \"d74e94faf585a6511b825684b07542bbdf47baab2f4f0bc80a198fc2c9d97709\": rpc error: code = NotFound desc = could not find container \"d74e94faf585a6511b825684b07542bbdf47baab2f4f0bc80a198fc2c9d97709\": container with ID starting with d74e94faf585a6511b825684b07542bbdf47baab2f4f0bc80a198fc2c9d97709 not found: ID does not exist" Nov 01 06:29:38.261580 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.261502 5040 scope.go:122] "RemoveContainer" containerID="c91c6d21f50b1bd24936f61867715b6115e4459b30f898aba28f2db276c632ef" Nov 01 06:29:38.261769 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:29:38.261750 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c91c6d21f50b1bd24936f61867715b6115e4459b30f898aba28f2db276c632ef\": container with ID starting with c91c6d21f50b1bd24936f61867715b6115e4459b30f898aba28f2db276c632ef not found: ID does not exist" containerID="c91c6d21f50b1bd24936f61867715b6115e4459b30f898aba28f2db276c632ef" Nov 01 06:29:38.261837 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.261779 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c91c6d21f50b1bd24936f61867715b6115e4459b30f898aba28f2db276c632ef"} err="failed to get container status \"c91c6d21f50b1bd24936f61867715b6115e4459b30f898aba28f2db276c632ef\": rpc error: code = NotFound desc = could not find container \"c91c6d21f50b1bd24936f61867715b6115e4459b30f898aba28f2db276c632ef\": container with ID starting with c91c6d21f50b1bd24936f61867715b6115e4459b30f898aba28f2db276c632ef not found: ID does not exist" Nov 01 06:29:38.408400 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.408320 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv"] Nov 01 06:29:38.415094 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:38.415066 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ce346503-openshift-default-74f467cf8c-vvwfv"] Nov 01 06:29:39.515073 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:39.515041 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61a726a8-a06e-4f27-95e9-bca8b6b78f6d" path="/var/lib/kubelet/pods/61a726a8-a06e-4f27-95e9-bca8b6b78f6d/volumes" Nov 01 06:29:39.515462 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:39.515446 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b1e622d-5b73-4c97-97eb-7bd89df19ca2" path="/var/lib/kubelet/pods/8b1e622d-5b73-4c97-97eb-7bd89df19ca2/volumes" Nov 01 06:29:39.974492 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:29:39.974381 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:29:54.378285 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:54.378251 5040 generic.go:356] "Generic (PLEG): container finished" podID="bb482d61-0774-4394-8c37-a564ef5b6838" containerID="a83bdbd5fe62ee118d2b1f3f325489a615d4b59532599da5a32e0f396fd47809" exitCode=0 Nov 01 06:29:54.378750 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:54.378299 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" event={"ID":"bb482d61-0774-4394-8c37-a564ef5b6838","Type":"ContainerDied","Data":"a83bdbd5fe62ee118d2b1f3f325489a615d4b59532599da5a32e0f396fd47809"} Nov 01 06:29:54.378750 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:54.378702 5040 scope.go:122] "RemoveContainer" containerID="a83bdbd5fe62ee118d2b1f3f325489a615d4b59532599da5a32e0f396fd47809" Nov 01 06:29:54.380309 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:54.380273 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:29:54.380388 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:54.380350 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:29:54.380431 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:54.380390 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:29:55.388797 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:55.388762 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" event={"ID":"bb482d61-0774-4394-8c37-a564ef5b6838","Type":"ContainerStarted","Data":"c2d5a44574e65916a9492d03badb46b392fdf88d0df62258518a4f9a0f759890"} Nov 01 06:29:55.878566 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:55.878530 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:29:55.878775 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:55.878751 5040 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 01 06:29:55.878822 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:55.878802 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" podUID="bb482d61-0774-4394-8c37-a564ef5b6838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 01 06:29:56.878152 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:56.878114 5040 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 01 06:29:56.878597 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:56.878166 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" podUID="bb482d61-0774-4394-8c37-a564ef5b6838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 01 06:29:57.877993 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:57.877955 5040 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 01 06:29:57.878179 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:57.878009 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" podUID="bb482d61-0774-4394-8c37-a564ef5b6838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 01 06:29:58.878544 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:58.878471 5040 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 01 06:29:58.878968 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:58.878573 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" podUID="bb482d61-0774-4394-8c37-a564ef5b6838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 01 06:29:59.878203 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:59.878163 5040 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 01 06:29:59.878392 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:29:59.878225 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" podUID="bb482d61-0774-4394-8c37-a564ef5b6838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 01 06:30:00.162890 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.162788 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891910-pt7jg"] Nov 01 06:30:00.163406 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.163366 5040 cpu_manager.go:414] "containerMap: removing container" podUID="61a726a8-a06e-4f27-95e9-bca8b6b78f6d" containerName="istio-proxy" Nov 01 06:30:00.163406 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.163386 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61a726a8-a06e-4f27-95e9-bca8b6b78f6d" containerName="istio-proxy" Nov 01 06:30:00.163581 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.163417 5040 cpu_manager.go:414] "containerMap: removing container" podUID="61a726a8-a06e-4f27-95e9-bca8b6b78f6d" containerName="istio-proxy" Nov 01 06:30:00.163581 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.163425 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61a726a8-a06e-4f27-95e9-bca8b6b78f6d" containerName="istio-proxy" Nov 01 06:30:00.163581 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.163568 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="61a726a8-a06e-4f27-95e9-bca8b6b78f6d" containerName="istio-proxy" Nov 01 06:30:00.163581 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.163580 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="61a726a8-a06e-4f27-95e9-bca8b6b78f6d" containerName="istio-proxy" Nov 01 06:30:00.167056 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.167031 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891910-6ngcr"] Nov 01 06:30:00.167197 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.167147 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891910-pt7jg" Nov 01 06:30:00.171287 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.171266 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891910-g2s6x"] Nov 01 06:30:00.171420 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.171395 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891910-6ngcr" Nov 01 06:30:00.174862 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.174843 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891910-g2s6x" Nov 01 06:30:00.220113 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.220072 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891910-4x68f"] Nov 01 06:30:00.224808 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.224784 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891910-cn5cf"] Nov 01 06:30:00.224932 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.224881 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891910-4x68f" Nov 01 06:30:00.229379 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.229356 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891910-cmplg"] Nov 01 06:30:00.229528 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.229455 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891910-cn5cf" Nov 01 06:30:00.233061 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.233040 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-745a18-29891910-gw4vg"] Nov 01 06:30:00.233176 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.233143 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-cmplg" Nov 01 06:30:00.236681 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.236661 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891910-6ptzx"] Nov 01 06:30:00.236903 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.236884 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-745a18-29891910-gw4vg" Nov 01 06:30:00.240577 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.240344 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891910-2vgzz"] Nov 01 06:30:00.240577 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.240467 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891910-6ptzx" Nov 01 06:30:00.244118 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.244089 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891910-kbwkk"] Nov 01 06:30:00.244225 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.244208 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891910-2vgzz" Nov 01 06:30:00.249799 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.249778 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891910-kbwkk" Nov 01 06:30:00.264392 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.264371 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:30:00.264549 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.264450 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:30:00.264549 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.264531 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:30:00.268369 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.268345 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891910-pt7jg"] Nov 01 06:30:00.271441 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.271419 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15896689-4508-43e1-95b2-df12c0853cd6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-3986bf-29891910-6ngcr\" (UID: \"15896689-4508-43e1-95b2-df12c0853cd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891910-6ngcr" Nov 01 06:30:00.271556 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.271464 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwclz\" (UniqueName: \"kubernetes.io/projected/aecbbf55-6343-4752-8e2b-0f3771c4df57-kube-api-access-lwclz\") pod \"maas-api-key-cleanup-e2e-api-b-745a18-29891910-gw4vg\" (UID: \"aecbbf55-6343-4752-8e2b-0f3771c4df57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-745a18-29891910-gw4vg" Nov 01 06:30:00.271614 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.271552 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da58795-e0ae-4db3-9774-0c8d997352a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891910-4x68f\" (UID: \"1da58795-e0ae-4db3-9774-0c8d997352a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891910-4x68f" Nov 01 06:30:00.271693 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.271641 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n5lz\" (UniqueName: \"kubernetes.io/projected/15896689-4508-43e1-95b2-df12c0853cd6-kube-api-access-2n5lz\") pod \"maas-api-key-cleanup-e2e-extmod-b-3986bf-29891910-6ngcr\" (UID: \"15896689-4508-43e1-95b2-df12c0853cd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891910-6ngcr" Nov 01 06:30:00.271693 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.271673 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4218688b-c54f-47cc-a11f-8fe7dde5cc48-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891910-cmplg\" (UID: \"4218688b-c54f-47cc-a11f-8fe7dde5cc48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-cmplg" Nov 01 06:30:00.271815 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.271709 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66ba8fe2-b2da-4281-aec7-7f2b8440e548-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891910-cn5cf\" (UID: \"66ba8fe2-b2da-4281-aec7-7f2b8440e548\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891910-cn5cf" Nov 01 06:30:00.271815 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.271737 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7c88e9-2475-4bf5-9c1a-426c1643312c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891910-g2s6x\" (UID: \"7c7c88e9-2475-4bf5-9c1a-426c1643312c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891910-g2s6x" Nov 01 06:30:00.271927 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.271858 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd27d740-cb57-479b-97ca-517a09d689b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-f73e89-29891910-2vgzz\" (UID: \"dd27d740-cb57-479b-97ca-517a09d689b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891910-2vgzz" Nov 01 06:30:00.271927 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.271886 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfhpp\" (UniqueName: \"kubernetes.io/projected/dd27d740-cb57-479b-97ca-517a09d689b3-kube-api-access-nfhpp\") pod \"maas-api-key-cleanup-e2e-worker-w4-f73e89-29891910-2vgzz\" (UID: \"dd27d740-cb57-479b-97ca-517a09d689b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891910-2vgzz" Nov 01 06:30:00.271927 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.271909 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891910-6ngcr"] Nov 01 06:30:00.272089 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.271956 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfxkb\" (UniqueName: \"kubernetes.io/projected/7c7c88e9-2475-4bf5-9c1a-426c1643312c-kube-api-access-dfxkb\") pod \"maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891910-g2s6x\" (UID: \"7c7c88e9-2475-4bf5-9c1a-426c1643312c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891910-g2s6x" Nov 01 06:30:00.272089 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.271995 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6gg5\" (UniqueName: \"kubernetes.io/projected/66ba8fe2-b2da-4281-aec7-7f2b8440e548-kube-api-access-v6gg5\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891910-cn5cf\" (UID: \"66ba8fe2-b2da-4281-aec7-7f2b8440e548\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891910-cn5cf" Nov 01 06:30:00.272089 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.272023 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m45sr\" (UniqueName: \"kubernetes.io/projected/4218688b-c54f-47cc-a11f-8fe7dde5cc48-kube-api-access-m45sr\") pod \"maas-api-key-cleanup-29891910-cmplg\" (UID: \"4218688b-c54f-47cc-a11f-8fe7dde5cc48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-cmplg" Nov 01 06:30:00.272279 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.272130 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1575b2f5-afcc-4d4e-8db0-98e419fced3f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891910-pt7jg\" (UID: \"1575b2f5-afcc-4d4e-8db0-98e419fced3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891910-pt7jg" Nov 01 06:30:00.272279 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.272161 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9stbz\" (UniqueName: \"kubernetes.io/projected/1575b2f5-afcc-4d4e-8db0-98e419fced3f-kube-api-access-9stbz\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891910-pt7jg\" (UID: \"1575b2f5-afcc-4d4e-8db0-98e419fced3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891910-pt7jg" Nov 01 06:30:00.272279 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.272189 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aecbbf55-6343-4752-8e2b-0f3771c4df57-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-745a18-29891910-gw4vg\" (UID: \"aecbbf55-6343-4752-8e2b-0f3771c4df57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-745a18-29891910-gw4vg" Nov 01 06:30:00.272279 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.272218 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbg2c\" (UniqueName: \"kubernetes.io/projected/98183e7b-70c1-4af0-9b64-9b4cba9424a1-kube-api-access-rbg2c\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d25db-29891910-6ptzx\" (UID: \"98183e7b-70c1-4af0-9b64-9b4cba9424a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891910-6ptzx" Nov 01 06:30:00.272279 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.272271 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98183e7b-70c1-4af0-9b64-9b4cba9424a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d25db-29891910-6ptzx\" (UID: \"98183e7b-70c1-4af0-9b64-9b4cba9424a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891910-6ptzx" Nov 01 06:30:00.272495 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.272347 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrwds\" (UniqueName: \"kubernetes.io/projected/a282a96e-d0fc-4d97-b2e1-35b8c593bcfc-kube-api-access-nrwds\") pod \"maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891910-kbwkk\" (UID: \"a282a96e-d0fc-4d97-b2e1-35b8c593bcfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891910-kbwkk" Nov 01 06:30:00.272495 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.272438 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a282a96e-d0fc-4d97-b2e1-35b8c593bcfc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891910-kbwkk\" (UID: \"a282a96e-d0fc-4d97-b2e1-35b8c593bcfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891910-kbwkk" Nov 01 06:30:00.272495 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.272462 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pjnd\" (UniqueName: \"kubernetes.io/projected/1da58795-e0ae-4db3-9774-0c8d997352a6-kube-api-access-6pjnd\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891910-4x68f\" (UID: \"1da58795-e0ae-4db3-9774-0c8d997352a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891910-4x68f" Nov 01 06:30:00.275228 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.275205 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891910-g2s6x"] Nov 01 06:30:00.278571 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.278550 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891910-cn5cf"] Nov 01 06:30:00.333002 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.330583 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891910-cmplg"] Nov 01 06:30:00.338740 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.336677 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-745a18-29891910-gw4vg"] Nov 01 06:30:00.372205 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.372171 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891910-2vgzz"] Nov 01 06:30:00.373871 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.373843 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n5lz\" (UniqueName: \"kubernetes.io/projected/15896689-4508-43e1-95b2-df12c0853cd6-kube-api-access-2n5lz\") pod \"maas-api-key-cleanup-e2e-extmod-b-3986bf-29891910-6ngcr\" (UID: \"15896689-4508-43e1-95b2-df12c0853cd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891910-6ngcr" Nov 01 06:30:00.373999 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.373880 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4218688b-c54f-47cc-a11f-8fe7dde5cc48-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891910-cmplg\" (UID: \"4218688b-c54f-47cc-a11f-8fe7dde5cc48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-cmplg" Nov 01 06:30:00.373999 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.373905 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66ba8fe2-b2da-4281-aec7-7f2b8440e548-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891910-cn5cf\" (UID: \"66ba8fe2-b2da-4281-aec7-7f2b8440e548\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891910-cn5cf" Nov 01 06:30:00.373999 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.373932 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7c88e9-2475-4bf5-9c1a-426c1643312c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891910-g2s6x\" (UID: \"7c7c88e9-2475-4bf5-9c1a-426c1643312c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891910-g2s6x" Nov 01 06:30:00.373999 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.373963 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd27d740-cb57-479b-97ca-517a09d689b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-f73e89-29891910-2vgzz\" (UID: \"dd27d740-cb57-479b-97ca-517a09d689b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891910-2vgzz" Nov 01 06:30:00.373999 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.373984 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfhpp\" (UniqueName: \"kubernetes.io/projected/dd27d740-cb57-479b-97ca-517a09d689b3-kube-api-access-nfhpp\") pod \"maas-api-key-cleanup-e2e-worker-w4-f73e89-29891910-2vgzz\" (UID: \"dd27d740-cb57-479b-97ca-517a09d689b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891910-2vgzz" Nov 01 06:30:00.374169 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.374002 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfxkb\" (UniqueName: \"kubernetes.io/projected/7c7c88e9-2475-4bf5-9c1a-426c1643312c-kube-api-access-dfxkb\") pod \"maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891910-g2s6x\" (UID: \"7c7c88e9-2475-4bf5-9c1a-426c1643312c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891910-g2s6x" Nov 01 06:30:00.374169 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.374022 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6gg5\" (UniqueName: \"kubernetes.io/projected/66ba8fe2-b2da-4281-aec7-7f2b8440e548-kube-api-access-v6gg5\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891910-cn5cf\" (UID: \"66ba8fe2-b2da-4281-aec7-7f2b8440e548\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891910-cn5cf" Nov 01 06:30:00.374169 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.374036 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m45sr\" (UniqueName: \"kubernetes.io/projected/4218688b-c54f-47cc-a11f-8fe7dde5cc48-kube-api-access-m45sr\") pod \"maas-api-key-cleanup-29891910-cmplg\" (UID: \"4218688b-c54f-47cc-a11f-8fe7dde5cc48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-cmplg" Nov 01 06:30:00.374169 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.374075 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1575b2f5-afcc-4d4e-8db0-98e419fced3f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891910-pt7jg\" (UID: \"1575b2f5-afcc-4d4e-8db0-98e419fced3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891910-pt7jg" Nov 01 06:30:00.374169 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.374095 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9stbz\" (UniqueName: \"kubernetes.io/projected/1575b2f5-afcc-4d4e-8db0-98e419fced3f-kube-api-access-9stbz\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891910-pt7jg\" (UID: \"1575b2f5-afcc-4d4e-8db0-98e419fced3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891910-pt7jg" Nov 01 06:30:00.374169 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.374112 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aecbbf55-6343-4752-8e2b-0f3771c4df57-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-745a18-29891910-gw4vg\" (UID: \"aecbbf55-6343-4752-8e2b-0f3771c4df57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-745a18-29891910-gw4vg" Nov 01 06:30:00.374169 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.374130 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbg2c\" (UniqueName: \"kubernetes.io/projected/98183e7b-70c1-4af0-9b64-9b4cba9424a1-kube-api-access-rbg2c\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d25db-29891910-6ptzx\" (UID: \"98183e7b-70c1-4af0-9b64-9b4cba9424a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891910-6ptzx" Nov 01 06:30:00.374169 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.374160 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98183e7b-70c1-4af0-9b64-9b4cba9424a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d25db-29891910-6ptzx\" (UID: \"98183e7b-70c1-4af0-9b64-9b4cba9424a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891910-6ptzx" Nov 01 06:30:00.374423 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.374185 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrwds\" (UniqueName: \"kubernetes.io/projected/a282a96e-d0fc-4d97-b2e1-35b8c593bcfc-kube-api-access-nrwds\") pod \"maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891910-kbwkk\" (UID: \"a282a96e-d0fc-4d97-b2e1-35b8c593bcfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891910-kbwkk" Nov 01 06:30:00.374423 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.374215 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a282a96e-d0fc-4d97-b2e1-35b8c593bcfc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891910-kbwkk\" (UID: \"a282a96e-d0fc-4d97-b2e1-35b8c593bcfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891910-kbwkk" Nov 01 06:30:00.374423 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.374230 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pjnd\" (UniqueName: \"kubernetes.io/projected/1da58795-e0ae-4db3-9774-0c8d997352a6-kube-api-access-6pjnd\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891910-4x68f\" (UID: \"1da58795-e0ae-4db3-9774-0c8d997352a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891910-4x68f" Nov 01 06:30:00.374423 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.374279 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15896689-4508-43e1-95b2-df12c0853cd6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-3986bf-29891910-6ngcr\" (UID: \"15896689-4508-43e1-95b2-df12c0853cd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891910-6ngcr" Nov 01 06:30:00.374423 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.374301 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwclz\" (UniqueName: \"kubernetes.io/projected/aecbbf55-6343-4752-8e2b-0f3771c4df57-kube-api-access-lwclz\") pod \"maas-api-key-cleanup-e2e-api-b-745a18-29891910-gw4vg\" (UID: \"aecbbf55-6343-4752-8e2b-0f3771c4df57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-745a18-29891910-gw4vg" Nov 01 06:30:00.374423 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.374316 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da58795-e0ae-4db3-9774-0c8d997352a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891910-4x68f\" (UID: \"1da58795-e0ae-4db3-9774-0c8d997352a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891910-4x68f" Nov 01 06:30:00.375194 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.375062 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da58795-e0ae-4db3-9774-0c8d997352a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891910-4x68f\" (UID: \"1da58795-e0ae-4db3-9774-0c8d997352a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891910-4x68f" Nov 01 06:30:00.377653 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.376032 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4218688b-c54f-47cc-a11f-8fe7dde5cc48-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891910-cmplg\" (UID: \"4218688b-c54f-47cc-a11f-8fe7dde5cc48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-cmplg" Nov 01 06:30:00.377653 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.376350 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15896689-4508-43e1-95b2-df12c0853cd6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-3986bf-29891910-6ngcr\" (UID: \"15896689-4508-43e1-95b2-df12c0853cd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891910-6ngcr" Nov 01 06:30:00.377653 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.376514 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98183e7b-70c1-4af0-9b64-9b4cba9424a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d25db-29891910-6ptzx\" (UID: \"98183e7b-70c1-4af0-9b64-9b4cba9424a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891910-6ptzx" Nov 01 06:30:00.377653 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.376636 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66ba8fe2-b2da-4281-aec7-7f2b8440e548-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891910-cn5cf\" (UID: \"66ba8fe2-b2da-4281-aec7-7f2b8440e548\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891910-cn5cf" Nov 01 06:30:00.377653 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.377016 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a282a96e-d0fc-4d97-b2e1-35b8c593bcfc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891910-kbwkk\" (UID: \"a282a96e-d0fc-4d97-b2e1-35b8c593bcfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891910-kbwkk" Nov 01 06:30:00.377653 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.377283 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891910-kbwkk"] Nov 01 06:30:00.377653 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.377607 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7c88e9-2475-4bf5-9c1a-426c1643312c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891910-g2s6x\" (UID: \"7c7c88e9-2475-4bf5-9c1a-426c1643312c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891910-g2s6x" Nov 01 06:30:00.378067 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.377678 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aecbbf55-6343-4752-8e2b-0f3771c4df57-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-745a18-29891910-gw4vg\" (UID: \"aecbbf55-6343-4752-8e2b-0f3771c4df57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-745a18-29891910-gw4vg" Nov 01 06:30:00.378067 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.377740 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1575b2f5-afcc-4d4e-8db0-98e419fced3f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891910-pt7jg\" (UID: \"1575b2f5-afcc-4d4e-8db0-98e419fced3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891910-pt7jg" Nov 01 06:30:00.378067 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.377748 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd27d740-cb57-479b-97ca-517a09d689b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-f73e89-29891910-2vgzz\" (UID: \"dd27d740-cb57-479b-97ca-517a09d689b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891910-2vgzz" Nov 01 06:30:00.424105 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.423599 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891910-4x68f"] Nov 01 06:30:00.430074 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.430030 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891910-6ptzx"] Nov 01 06:30:00.440521 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.440403 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m45sr\" (UniqueName: \"kubernetes.io/projected/4218688b-c54f-47cc-a11f-8fe7dde5cc48-kube-api-access-m45sr\") pod \"maas-api-key-cleanup-29891910-cmplg\" (UID: \"4218688b-c54f-47cc-a11f-8fe7dde5cc48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-cmplg" Nov 01 06:30:00.441599 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.441510 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwclz\" (UniqueName: \"kubernetes.io/projected/aecbbf55-6343-4752-8e2b-0f3771c4df57-kube-api-access-lwclz\") pod \"maas-api-key-cleanup-e2e-api-b-745a18-29891910-gw4vg\" (UID: \"aecbbf55-6343-4752-8e2b-0f3771c4df57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-745a18-29891910-gw4vg" Nov 01 06:30:00.444469 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.443108 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n5lz\" (UniqueName: \"kubernetes.io/projected/15896689-4508-43e1-95b2-df12c0853cd6-kube-api-access-2n5lz\") pod \"maas-api-key-cleanup-e2e-extmod-b-3986bf-29891910-6ngcr\" (UID: \"15896689-4508-43e1-95b2-df12c0853cd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891910-6ngcr" Nov 01 06:30:00.444748 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.443700 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbg2c\" (UniqueName: \"kubernetes.io/projected/98183e7b-70c1-4af0-9b64-9b4cba9424a1-kube-api-access-rbg2c\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d25db-29891910-6ptzx\" (UID: \"98183e7b-70c1-4af0-9b64-9b4cba9424a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891910-6ptzx" Nov 01 06:30:00.445428 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.444287 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfhpp\" (UniqueName: \"kubernetes.io/projected/dd27d740-cb57-479b-97ca-517a09d689b3-kube-api-access-nfhpp\") pod \"maas-api-key-cleanup-e2e-worker-w4-f73e89-29891910-2vgzz\" (UID: \"dd27d740-cb57-479b-97ca-517a09d689b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891910-2vgzz" Nov 01 06:30:00.446165 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.446146 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9stbz\" (UniqueName: \"kubernetes.io/projected/1575b2f5-afcc-4d4e-8db0-98e419fced3f-kube-api-access-9stbz\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891910-pt7jg\" (UID: \"1575b2f5-afcc-4d4e-8db0-98e419fced3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891910-pt7jg" Nov 01 06:30:00.447516 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.447120 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6gg5\" (UniqueName: \"kubernetes.io/projected/66ba8fe2-b2da-4281-aec7-7f2b8440e548-kube-api-access-v6gg5\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891910-cn5cf\" (UID: \"66ba8fe2-b2da-4281-aec7-7f2b8440e548\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891910-cn5cf" Nov 01 06:30:00.454897 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.454845 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrwds\" (UniqueName: \"kubernetes.io/projected/a282a96e-d0fc-4d97-b2e1-35b8c593bcfc-kube-api-access-nrwds\") pod \"maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891910-kbwkk\" (UID: \"a282a96e-d0fc-4d97-b2e1-35b8c593bcfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891910-kbwkk" Nov 01 06:30:00.458215 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.458146 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfxkb\" (UniqueName: \"kubernetes.io/projected/7c7c88e9-2475-4bf5-9c1a-426c1643312c-kube-api-access-dfxkb\") pod \"maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891910-g2s6x\" (UID: \"7c7c88e9-2475-4bf5-9c1a-426c1643312c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891910-g2s6x" Nov 01 06:30:00.461172 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.460599 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pjnd\" (UniqueName: \"kubernetes.io/projected/1da58795-e0ae-4db3-9774-0c8d997352a6-kube-api-access-6pjnd\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891910-4x68f\" (UID: \"1da58795-e0ae-4db3-9774-0c8d997352a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891910-4x68f" Nov 01 06:30:00.482854 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.482828 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891910-pt7jg" Nov 01 06:30:00.490813 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.490785 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891910-6ngcr" Nov 01 06:30:00.521351 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.521295 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891910-g2s6x" Nov 01 06:30:00.540658 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.540625 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891910-4x68f" Nov 01 06:30:00.554622 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.551734 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891910-cn5cf" Nov 01 06:30:00.559114 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.558700 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-cmplg" Nov 01 06:30:00.575066 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.575032 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-745a18-29891910-gw4vg" Nov 01 06:30:00.575974 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.575828 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891910-6ptzx" Nov 01 06:30:00.588528 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.585610 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891910-2vgzz" Nov 01 06:30:00.593737 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.593518 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891910-kbwkk" Nov 01 06:30:00.887613 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.883796 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:30:00.887613 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.884346 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:30:00.887613 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.884960 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:30:00.887613 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:30:00.886993 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1575b2f5_afcc_4d4e_8db0_98e419fced3f.slice/crio-a2a55072832f368ebae2104c1995b7b6bea08b6a95fdfbcd06a17baceb03c418 WatchSource:0}: Error finding container a2a55072832f368ebae2104c1995b7b6bea08b6a95fdfbcd06a17baceb03c418: Status 404 returned error can't find the container with id a2a55072832f368ebae2104c1995b7b6bea08b6a95fdfbcd06a17baceb03c418 Nov 01 06:30:00.887613 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.887264 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891910-pt7jg"] Nov 01 06:30:00.934800 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.934770 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891910-6ngcr"] Nov 01 06:30:00.955992 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:30:00.955963 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15896689_4508_43e1_95b2_df12c0853cd6.slice/crio-77d02b08f87b8aa1374b76e41f9cf61e7cbe1993a18546d1865917338f488c44 WatchSource:0}: Error finding container 77d02b08f87b8aa1374b76e41f9cf61e7cbe1993a18546d1865917338f488c44: Status 404 returned error can't find the container with id 77d02b08f87b8aa1374b76e41f9cf61e7cbe1993a18546d1865917338f488c44 Nov 01 06:30:00.991332 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:00.991304 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891910-4x68f"] Nov 01 06:30:01.123166 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:01.123069 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891910-2vgzz"] Nov 01 06:30:01.128422 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:30:01.128394 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda282a96e_d0fc_4d97_b2e1_35b8c593bcfc.slice/crio-de53d7e9af4308402b9f8378d58f17ea28be1d6c42b8e4e1bcfd1d80ae6e854d WatchSource:0}: Error finding container de53d7e9af4308402b9f8378d58f17ea28be1d6c42b8e4e1bcfd1d80ae6e854d: Status 404 returned error can't find the container with id de53d7e9af4308402b9f8378d58f17ea28be1d6c42b8e4e1bcfd1d80ae6e854d Nov 01 06:30:01.128546 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:01.128424 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891910-kbwkk"] Nov 01 06:30:01.148767 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:01.148735 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891910-g2s6x"] Nov 01 06:30:01.167392 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:30:01.167361 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4218688b_c54f_47cc_a11f_8fe7dde5cc48.slice/crio-37dc60be50ff8ee0ac23f679394c72660939327ec54a981bbd5d67979d118a72 WatchSource:0}: Error finding container 37dc60be50ff8ee0ac23f679394c72660939327ec54a981bbd5d67979d118a72: Status 404 returned error can't find the container with id 37dc60be50ff8ee0ac23f679394c72660939327ec54a981bbd5d67979d118a72 Nov 01 06:30:01.170575 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:01.170548 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891910-cmplg"] Nov 01 06:30:01.293863 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:01.293771 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891910-6ptzx"] Nov 01 06:30:01.298328 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:01.298297 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891910-cn5cf"] Nov 01 06:30:01.300641 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:30:01.300612 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98183e7b_70c1_4af0_9b64_9b4cba9424a1.slice/crio-b61f4c4160e819d3fa44168fb73bd1ef497d63e1b4fa1af516264600e05e05f4 WatchSource:0}: Error finding container b61f4c4160e819d3fa44168fb73bd1ef497d63e1b4fa1af516264600e05e05f4: Status 404 returned error can't find the container with id b61f4c4160e819d3fa44168fb73bd1ef497d63e1b4fa1af516264600e05e05f4 Nov 01 06:30:01.302352 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:01.302322 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-745a18-29891910-gw4vg"] Nov 01 06:30:01.304026 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:30:01.304001 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaecbbf55_6343_4752_8e2b_0f3771c4df57.slice/crio-a0d1891b9b81c99d803f97e17c4b9a92066297b5ba6d2aa165ce9d70eb3c6235 WatchSource:0}: Error finding container a0d1891b9b81c99d803f97e17c4b9a92066297b5ba6d2aa165ce9d70eb3c6235: Status 404 returned error can't find the container with id a0d1891b9b81c99d803f97e17c4b9a92066297b5ba6d2aa165ce9d70eb3c6235 Nov 01 06:30:01.313275 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:30:01.313252 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ba8fe2_b2da_4281_aec7_7f2b8440e548.slice/crio-cb8d9717cbc13a9667e37537f53d997938b37c2b63088b2f5bec7b5ffad82692 WatchSource:0}: Error finding container cb8d9717cbc13a9667e37537f53d997938b37c2b63088b2f5bec7b5ffad82692: Status 404 returned error can't find the container with id cb8d9717cbc13a9667e37537f53d997938b37c2b63088b2f5bec7b5ffad82692 Nov 01 06:30:01.480162 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:01.480126 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891910-4x68f" event={"ID":"1da58795-e0ae-4db3-9774-0c8d997352a6","Type":"ContainerStarted","Data":"7e0d2866cb201ce7b14c378c0e9eff71639152daaea426eeb525b0e75486840a"} Nov 01 06:30:01.481617 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:01.481591 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891910-2vgzz" event={"ID":"dd27d740-cb57-479b-97ca-517a09d689b3","Type":"ContainerStarted","Data":"c7c2be49bee8858ce7dc14fa3316978c62233e8a145b7dd943fdf7993502a073"} Nov 01 06:30:01.482997 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:01.482971 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-745a18-29891910-gw4vg" event={"ID":"aecbbf55-6343-4752-8e2b-0f3771c4df57","Type":"ContainerStarted","Data":"a0d1891b9b81c99d803f97e17c4b9a92066297b5ba6d2aa165ce9d70eb3c6235"} Nov 01 06:30:01.484334 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:01.484303 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891910-g2s6x" event={"ID":"7c7c88e9-2475-4bf5-9c1a-426c1643312c","Type":"ContainerStarted","Data":"c6ab5b64e3aae7b6175da312733ac6ba8a60228839cbb05bb4ee0e6b66ceabc5"} Nov 01 06:30:01.485793 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:01.485770 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-cmplg" event={"ID":"4218688b-c54f-47cc-a11f-8fe7dde5cc48","Type":"ContainerStarted","Data":"37dc60be50ff8ee0ac23f679394c72660939327ec54a981bbd5d67979d118a72"} Nov 01 06:30:01.487163 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:01.487141 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891910-kbwkk" event={"ID":"a282a96e-d0fc-4d97-b2e1-35b8c593bcfc","Type":"ContainerStarted","Data":"de53d7e9af4308402b9f8378d58f17ea28be1d6c42b8e4e1bcfd1d80ae6e854d"} Nov 01 06:30:01.489204 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:01.489174 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891910-6ngcr" event={"ID":"15896689-4508-43e1-95b2-df12c0853cd6","Type":"ContainerStarted","Data":"77d02b08f87b8aa1374b76e41f9cf61e7cbe1993a18546d1865917338f488c44"} Nov 01 06:30:01.490519 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:01.490498 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891910-pt7jg" event={"ID":"1575b2f5-afcc-4d4e-8db0-98e419fced3f","Type":"ContainerStarted","Data":"a2a55072832f368ebae2104c1995b7b6bea08b6a95fdfbcd06a17baceb03c418"} Nov 01 06:30:01.492038 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:01.492014 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891910-6ptzx" event={"ID":"98183e7b-70c1-4af0-9b64-9b4cba9424a1","Type":"ContainerStarted","Data":"b61f4c4160e819d3fa44168fb73bd1ef497d63e1b4fa1af516264600e05e05f4"} Nov 01 06:30:01.493398 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:01.493376 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891910-cn5cf" event={"ID":"66ba8fe2-b2da-4281-aec7-7f2b8440e548","Type":"ContainerStarted","Data":"cb8d9717cbc13a9667e37537f53d997938b37c2b63088b2f5bec7b5ffad82692"} Nov 01 06:30:02.510287 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:02.510246 5040 generic.go:356] "Generic (PLEG): container finished" podID="4218688b-c54f-47cc-a11f-8fe7dde5cc48" containerID="56ccee39e001011b84e90fa72a770f19db0f3519451bd23644b45354ccde32dd" exitCode=60 Nov 01 06:30:02.510990 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:02.510408 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-cmplg" event={"ID":"4218688b-c54f-47cc-a11f-8fe7dde5cc48","Type":"ContainerDied","Data":"56ccee39e001011b84e90fa72a770f19db0f3519451bd23644b45354ccde32dd"} Nov 01 06:30:02.511602 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:02.511565 5040 scope.go:122] "RemoveContainer" containerID="56ccee39e001011b84e90fa72a770f19db0f3519451bd23644b45354ccde32dd" Nov 01 06:30:02.517038 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:02.517006 5040 generic.go:356] "Generic (PLEG): container finished" podID="a282a96e-d0fc-4d97-b2e1-35b8c593bcfc" containerID="14502ae2a339def2956a977bf5d60bc4cb0b29ac992cab36ac60568429bfaeb5" exitCode=60 Nov 01 06:30:02.517165 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:02.517085 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891910-kbwkk" event={"ID":"a282a96e-d0fc-4d97-b2e1-35b8c593bcfc","Type":"ContainerDied","Data":"14502ae2a339def2956a977bf5d60bc4cb0b29ac992cab36ac60568429bfaeb5"} Nov 01 06:30:02.517417 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:02.517398 5040 scope.go:122] "RemoveContainer" containerID="14502ae2a339def2956a977bf5d60bc4cb0b29ac992cab36ac60568429bfaeb5" Nov 01 06:30:02.519344 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:02.519316 5040 generic.go:356] "Generic (PLEG): container finished" podID="15896689-4508-43e1-95b2-df12c0853cd6" containerID="98cd44a7defe0e020859b1f92c625185f1eb4cfffcdb6156ecf61509ce935f27" exitCode=60 Nov 01 06:30:02.519508 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:02.519461 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891910-6ngcr" event={"ID":"15896689-4508-43e1-95b2-df12c0853cd6","Type":"ContainerDied","Data":"98cd44a7defe0e020859b1f92c625185f1eb4cfffcdb6156ecf61509ce935f27"} Nov 01 06:30:02.519928 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:02.519909 5040 scope.go:122] "RemoveContainer" containerID="98cd44a7defe0e020859b1f92c625185f1eb4cfffcdb6156ecf61509ce935f27" Nov 01 06:30:02.522008 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:02.521983 5040 generic.go:356] "Generic (PLEG): container finished" podID="1575b2f5-afcc-4d4e-8db0-98e419fced3f" containerID="93fb7a4b3ae421cf7ef6a428c86c5709f800893b481500283d15d401d743af5e" exitCode=60 Nov 01 06:30:02.522164 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:02.522146 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891910-pt7jg" event={"ID":"1575b2f5-afcc-4d4e-8db0-98e419fced3f","Type":"ContainerDied","Data":"93fb7a4b3ae421cf7ef6a428c86c5709f800893b481500283d15d401d743af5e"} Nov 01 06:30:02.522694 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:02.522670 5040 scope.go:122] "RemoveContainer" containerID="93fb7a4b3ae421cf7ef6a428c86c5709f800893b481500283d15d401d743af5e" Nov 01 06:30:02.524385 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:02.524364 5040 generic.go:356] "Generic (PLEG): container finished" podID="98183e7b-70c1-4af0-9b64-9b4cba9424a1" containerID="8ca6788518aac4758cfe552a22fa3d4d04eff8852c61485a06e1b6bf14edfea9" exitCode=60 Nov 01 06:30:02.524512 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:02.524438 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891910-6ptzx" event={"ID":"98183e7b-70c1-4af0-9b64-9b4cba9424a1","Type":"ContainerDied","Data":"8ca6788518aac4758cfe552a22fa3d4d04eff8852c61485a06e1b6bf14edfea9"} Nov 01 06:30:02.524673 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:02.524653 5040 scope.go:122] "RemoveContainer" containerID="8ca6788518aac4758cfe552a22fa3d4d04eff8852c61485a06e1b6bf14edfea9" Nov 01 06:30:02.526111 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:02.526090 5040 generic.go:356] "Generic (PLEG): container finished" podID="66ba8fe2-b2da-4281-aec7-7f2b8440e548" containerID="4b8f77324e596a2dc3958a77c2b213556e62800aa37f05ac279239010a42e428" exitCode=60 Nov 01 06:30:02.526247 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:02.526228 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891910-cn5cf" event={"ID":"66ba8fe2-b2da-4281-aec7-7f2b8440e548","Type":"ContainerDied","Data":"4b8f77324e596a2dc3958a77c2b213556e62800aa37f05ac279239010a42e428"} Nov 01 06:30:02.526366 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:02.526340 5040 scope.go:122] "RemoveContainer" containerID="4b8f77324e596a2dc3958a77c2b213556e62800aa37f05ac279239010a42e428" Nov 01 06:30:02.528683 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:02.528664 5040 generic.go:356] "Generic (PLEG): container finished" podID="1da58795-e0ae-4db3-9774-0c8d997352a6" containerID="80c7368a419e838db8cb8ef46fc55763f9c9eaff5d725c2e864a64972d9e6173" exitCode=60 Nov 01 06:30:02.528916 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:02.528751 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891910-4x68f" event={"ID":"1da58795-e0ae-4db3-9774-0c8d997352a6","Type":"ContainerDied","Data":"80c7368a419e838db8cb8ef46fc55763f9c9eaff5d725c2e864a64972d9e6173"} Nov 01 06:30:02.529027 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:02.528969 5040 scope.go:122] "RemoveContainer" containerID="80c7368a419e838db8cb8ef46fc55763f9c9eaff5d725c2e864a64972d9e6173" Nov 01 06:30:02.534212 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:02.534190 5040 generic.go:356] "Generic (PLEG): container finished" podID="dd27d740-cb57-479b-97ca-517a09d689b3" containerID="8e973a0aa182996ea440374ea310c903bc335487f3cd4d9df304897ab5cdbd16" exitCode=60 Nov 01 06:30:02.534308 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:02.534257 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891910-2vgzz" event={"ID":"dd27d740-cb57-479b-97ca-517a09d689b3","Type":"ContainerDied","Data":"8e973a0aa182996ea440374ea310c903bc335487f3cd4d9df304897ab5cdbd16"} Nov 01 06:30:02.534470 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:02.534455 5040 scope.go:122] "RemoveContainer" containerID="8e973a0aa182996ea440374ea310c903bc335487f3cd4d9df304897ab5cdbd16" Nov 01 06:30:02.535985 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:02.535965 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-745a18-29891910-gw4vg" event={"ID":"aecbbf55-6343-4752-8e2b-0f3771c4df57","Type":"ContainerStarted","Data":"10d10320f776c0a66e97594767d20bfd1174678f2cd2854f6ad18a2c3bec3825"} Nov 01 06:30:02.536162 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:02.536147 5040 scope.go:122] "RemoveContainer" containerID="10d10320f776c0a66e97594767d20bfd1174678f2cd2854f6ad18a2c3bec3825" Nov 01 06:30:02.537901 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:02.537881 5040 generic.go:356] "Generic (PLEG): container finished" podID="7c7c88e9-2475-4bf5-9c1a-426c1643312c" containerID="c1e1ba7da29e26419fa4675849bd4e8b52a4601cdad830ec7b214772bdd85366" exitCode=60 Nov 01 06:30:02.538774 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:02.538752 5040 scope.go:122] "RemoveContainer" containerID="c1e1ba7da29e26419fa4675849bd4e8b52a4601cdad830ec7b214772bdd85366" Nov 01 06:30:02.539077 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:02.539051 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891910-g2s6x" event={"ID":"7c7c88e9-2475-4bf5-9c1a-426c1643312c","Type":"ContainerDied","Data":"c1e1ba7da29e26419fa4675849bd4e8b52a4601cdad830ec7b214772bdd85366"} Nov 01 06:30:03.548282 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:03.548253 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891910-4x68f" event={"ID":"1da58795-e0ae-4db3-9774-0c8d997352a6","Type":"ContainerStarted","Data":"3982ea529b948eeb5e1da63a5870cdd186b028fd73ed65c3e5b1e18a60d7d403"} Nov 01 06:30:03.550095 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:03.550072 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891910-2vgzz" event={"ID":"dd27d740-cb57-479b-97ca-517a09d689b3","Type":"ContainerStarted","Data":"8182279e7a331feb0fd1f9194a69a35536d4f72c76c84748b061f6ba5e0990a5"} Nov 01 06:30:03.551882 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:03.551859 5040 generic.go:356] "Generic (PLEG): container finished" podID="aecbbf55-6343-4752-8e2b-0f3771c4df57" containerID="10d10320f776c0a66e97594767d20bfd1174678f2cd2854f6ad18a2c3bec3825" exitCode=60 Nov 01 06:30:03.551986 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:03.551922 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-745a18-29891910-gw4vg" event={"ID":"aecbbf55-6343-4752-8e2b-0f3771c4df57","Type":"ContainerDied","Data":"10d10320f776c0a66e97594767d20bfd1174678f2cd2854f6ad18a2c3bec3825"} Nov 01 06:30:03.551986 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:03.551958 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-745a18-29891910-gw4vg" event={"ID":"aecbbf55-6343-4752-8e2b-0f3771c4df57","Type":"ContainerStarted","Data":"8017e9c7e75da8416033457984dec78d13afe6e9ff7ffd6df112464305d16b1b"} Nov 01 06:30:03.553730 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:03.553665 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891910-g2s6x" event={"ID":"7c7c88e9-2475-4bf5-9c1a-426c1643312c","Type":"ContainerStarted","Data":"ea3f072afdebdedf1a5d87af92c1c185d5df8b08a114f37a2b410f473d9e0d39"} Nov 01 06:30:03.555378 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:03.555357 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-cmplg" event={"ID":"4218688b-c54f-47cc-a11f-8fe7dde5cc48","Type":"ContainerStarted","Data":"9844b6f27212b2bc32183218acd0c59ae1eae92c5643b57a19551a4e5086af13"} Nov 01 06:30:03.557208 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:03.557187 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891910-kbwkk" event={"ID":"a282a96e-d0fc-4d97-b2e1-35b8c593bcfc","Type":"ContainerStarted","Data":"2182bd2fd02855b1175c942c5ccf3640415d0bb9ab0eb0f7af8c0c47327495e7"} Nov 01 06:30:03.559007 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:03.558988 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891910-6ngcr" event={"ID":"15896689-4508-43e1-95b2-df12c0853cd6","Type":"ContainerStarted","Data":"8076a746a19b6ea48ae69f8b6cad719b0d910f0a0ae64ab20eab477efc917449"} Nov 01 06:30:03.560846 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:03.560825 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891910-pt7jg" event={"ID":"1575b2f5-afcc-4d4e-8db0-98e419fced3f","Type":"ContainerStarted","Data":"0df498437e0b0b598a72e861d5c1ebe3ec45deb362bf5e5e3a30285c7c8551bd"} Nov 01 06:30:03.562690 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:03.562670 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891910-6ptzx" event={"ID":"98183e7b-70c1-4af0-9b64-9b4cba9424a1","Type":"ContainerStarted","Data":"39612278cd707ab2a21a2010ed477573cee16a1f8ba41307c27ac5dc15d86eb1"} Nov 01 06:30:03.564401 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:03.564382 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891910-cn5cf" event={"ID":"66ba8fe2-b2da-4281-aec7-7f2b8440e548","Type":"ContainerStarted","Data":"990141bde89912ea666dd6c6d989fb5c9816bb256ee4aafb49df44e89f14618c"} Nov 01 06:30:03.837254 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:03.837202 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891910-g2s6x" podStartSLOduration=3.83718801 podStartE2EDuration="3.83718801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:03.835661461 +0000 UTC m=+2682.490778628" watchObservedRunningTime="2026-11-01 06:30:03.83718801 +0000 UTC m=+2682.492305177" Nov 01 06:30:03.951687 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:03.951638 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891910-2vgzz" podStartSLOduration=3.9516223740000003 podStartE2EDuration="3.951622374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:03.950968307 +0000 UTC m=+2682.606085474" watchObservedRunningTime="2026-11-01 06:30:03.951622374 +0000 UTC m=+2682.606739541" Nov 01 06:30:04.069356 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.069308 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891910-4x68f" podStartSLOduration=4.069295639 podStartE2EDuration="4.069295639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:04.068095284 +0000 UTC m=+2682.723212451" watchObservedRunningTime="2026-11-01 06:30:04.069295639 +0000 UTC m=+2682.724412845" Nov 01 06:30:04.185985 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.185937 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-cmplg" podStartSLOduration=4.185922559 podStartE2EDuration="4.185922559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:04.183732694 +0000 UTC m=+2682.838849855" watchObservedRunningTime="2026-11-01 06:30:04.185922559 +0000 UTC m=+2682.841039726" Nov 01 06:30:04.315472 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.315340 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-745a18-29891910-gw4vg" podStartSLOduration=4.315320036 podStartE2EDuration="4.315320036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:04.301851688 +0000 UTC m=+2682.956968859" watchObservedRunningTime="2026-11-01 06:30:04.315320036 +0000 UTC m=+2682.970437206" Nov 01 06:30:04.420854 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.420800 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891910-pt7jg" podStartSLOduration=4.4207869859999995 podStartE2EDuration="4.420786986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:04.419625184 +0000 UTC m=+2683.074742352" watchObservedRunningTime="2026-11-01 06:30:04.420786986 +0000 UTC m=+2683.075904152" Nov 01 06:30:04.536384 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.536329 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891910-6ptzx" podStartSLOduration=4.536316013 podStartE2EDuration="4.536316013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:04.535641507 +0000 UTC m=+2683.190758675" watchObservedRunningTime="2026-11-01 06:30:04.536316013 +0000 UTC m=+2683.191433179" Nov 01 06:30:04.580606 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.580522 5040 generic.go:356] "Generic (PLEG): container finished" podID="1da58795-e0ae-4db3-9774-0c8d997352a6" containerID="3982ea529b948eeb5e1da63a5870cdd186b028fd73ed65c3e5b1e18a60d7d403" exitCode=60 Nov 01 06:30:04.581030 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.580603 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891910-4x68f" event={"ID":"1da58795-e0ae-4db3-9774-0c8d997352a6","Type":"ContainerDied","Data":"3982ea529b948eeb5e1da63a5870cdd186b028fd73ed65c3e5b1e18a60d7d403"} Nov 01 06:30:04.581030 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.580695 5040 scope.go:122] "RemoveContainer" containerID="80c7368a419e838db8cb8ef46fc55763f9c9eaff5d725c2e864a64972d9e6173" Nov 01 06:30:04.581030 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.580797 5040 scope.go:122] "RemoveContainer" containerID="3982ea529b948eeb5e1da63a5870cdd186b028fd73ed65c3e5b1e18a60d7d403" Nov 01 06:30:04.581179 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:30:04.581071 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891910-4x68f_odh-ai-gateway-infra(1da58795-e0ae-4db3-9774-0c8d997352a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891910-4x68f" podUID="1da58795-e0ae-4db3-9774-0c8d997352a6" Nov 01 06:30:04.582736 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.582714 5040 generic.go:356] "Generic (PLEG): container finished" podID="dd27d740-cb57-479b-97ca-517a09d689b3" containerID="8182279e7a331feb0fd1f9194a69a35536d4f72c76c84748b061f6ba5e0990a5" exitCode=60 Nov 01 06:30:04.582861 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.582779 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891910-2vgzz" event={"ID":"dd27d740-cb57-479b-97ca-517a09d689b3","Type":"ContainerDied","Data":"8182279e7a331feb0fd1f9194a69a35536d4f72c76c84748b061f6ba5e0990a5"} Nov 01 06:30:04.583045 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.583031 5040 scope.go:122] "RemoveContainer" containerID="8182279e7a331feb0fd1f9194a69a35536d4f72c76c84748b061f6ba5e0990a5" Nov 01 06:30:04.583247 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:30:04.583227 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-f73e89-29891910-2vgzz_odh-ai-gateway-infra(dd27d740-cb57-479b-97ca-517a09d689b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891910-2vgzz" podUID="dd27d740-cb57-479b-97ca-517a09d689b3" Nov 01 06:30:04.584983 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.584964 5040 generic.go:356] "Generic (PLEG): container finished" podID="aecbbf55-6343-4752-8e2b-0f3771c4df57" containerID="8017e9c7e75da8416033457984dec78d13afe6e9ff7ffd6df112464305d16b1b" exitCode=60 Nov 01 06:30:04.585082 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.585016 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-745a18-29891910-gw4vg" event={"ID":"aecbbf55-6343-4752-8e2b-0f3771c4df57","Type":"ContainerDied","Data":"8017e9c7e75da8416033457984dec78d13afe6e9ff7ffd6df112464305d16b1b"} Nov 01 06:30:04.585243 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.585229 5040 scope.go:122] "RemoveContainer" containerID="8017e9c7e75da8416033457984dec78d13afe6e9ff7ffd6df112464305d16b1b" Nov 01 06:30:04.585402 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:30:04.585387 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-745a18-29891910-gw4vg_odh-ai-gateway-infra(aecbbf55-6343-4752-8e2b-0f3771c4df57)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-745a18-29891910-gw4vg" podUID="aecbbf55-6343-4752-8e2b-0f3771c4df57" Nov 01 06:30:04.587136 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.587117 5040 generic.go:356] "Generic (PLEG): container finished" podID="7c7c88e9-2475-4bf5-9c1a-426c1643312c" containerID="ea3f072afdebdedf1a5d87af92c1c185d5df8b08a114f37a2b410f473d9e0d39" exitCode=60 Nov 01 06:30:04.587235 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.587182 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891910-g2s6x" event={"ID":"7c7c88e9-2475-4bf5-9c1a-426c1643312c","Type":"ContainerDied","Data":"ea3f072afdebdedf1a5d87af92c1c185d5df8b08a114f37a2b410f473d9e0d39"} Nov 01 06:30:04.587398 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.587385 5040 scope.go:122] "RemoveContainer" containerID="ea3f072afdebdedf1a5d87af92c1c185d5df8b08a114f37a2b410f473d9e0d39" Nov 01 06:30:04.587569 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:30:04.587552 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891910-g2s6x_odh-ai-gateway-infra(7c7c88e9-2475-4bf5-9c1a-426c1643312c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891910-g2s6x" podUID="7c7c88e9-2475-4bf5-9c1a-426c1643312c" Nov 01 06:30:04.589678 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.589655 5040 generic.go:356] "Generic (PLEG): container finished" podID="4218688b-c54f-47cc-a11f-8fe7dde5cc48" containerID="9844b6f27212b2bc32183218acd0c59ae1eae92c5643b57a19551a4e5086af13" exitCode=60 Nov 01 06:30:04.589770 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.589757 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-cmplg" event={"ID":"4218688b-c54f-47cc-a11f-8fe7dde5cc48","Type":"ContainerDied","Data":"9844b6f27212b2bc32183218acd0c59ae1eae92c5643b57a19551a4e5086af13"} Nov 01 06:30:04.590051 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.590037 5040 scope.go:122] "RemoveContainer" containerID="9844b6f27212b2bc32183218acd0c59ae1eae92c5643b57a19551a4e5086af13" Nov 01 06:30:04.590249 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:30:04.590230 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891910-cmplg_odh-ai-gateway-infra(4218688b-c54f-47cc-a11f-8fe7dde5cc48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-cmplg" podUID="4218688b-c54f-47cc-a11f-8fe7dde5cc48" Nov 01 06:30:04.592187 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.592166 5040 generic.go:356] "Generic (PLEG): container finished" podID="a282a96e-d0fc-4d97-b2e1-35b8c593bcfc" containerID="2182bd2fd02855b1175c942c5ccf3640415d0bb9ab0eb0f7af8c0c47327495e7" exitCode=60 Nov 01 06:30:04.592277 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.592250 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891910-kbwkk" event={"ID":"a282a96e-d0fc-4d97-b2e1-35b8c593bcfc","Type":"ContainerDied","Data":"2182bd2fd02855b1175c942c5ccf3640415d0bb9ab0eb0f7af8c0c47327495e7"} Nov 01 06:30:04.592435 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.592418 5040 scope.go:122] "RemoveContainer" containerID="2182bd2fd02855b1175c942c5ccf3640415d0bb9ab0eb0f7af8c0c47327495e7" Nov 01 06:30:04.592677 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:30:04.592650 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891910-kbwkk_odh-ai-gateway-infra(a282a96e-d0fc-4d97-b2e1-35b8c593bcfc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891910-kbwkk" podUID="a282a96e-d0fc-4d97-b2e1-35b8c593bcfc" Nov 01 06:30:04.618873 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.618724 5040 scope.go:122] "RemoveContainer" containerID="8e973a0aa182996ea440374ea310c903bc335487f3cd4d9df304897ab5cdbd16" Nov 01 06:30:04.619385 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.619349 5040 generic.go:356] "Generic (PLEG): container finished" podID="15896689-4508-43e1-95b2-df12c0853cd6" containerID="8076a746a19b6ea48ae69f8b6cad719b0d910f0a0ae64ab20eab477efc917449" exitCode=60 Nov 01 06:30:04.619710 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.619681 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891910-6ngcr" event={"ID":"15896689-4508-43e1-95b2-df12c0853cd6","Type":"ContainerDied","Data":"8076a746a19b6ea48ae69f8b6cad719b0d910f0a0ae64ab20eab477efc917449"} Nov 01 06:30:04.619912 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.619886 5040 scope.go:122] "RemoveContainer" containerID="8076a746a19b6ea48ae69f8b6cad719b0d910f0a0ae64ab20eab477efc917449" Nov 01 06:30:04.620271 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:30:04.620244 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-3986bf-29891910-6ngcr_odh-ai-gateway-infra(15896689-4508-43e1-95b2-df12c0853cd6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891910-6ngcr" podUID="15896689-4508-43e1-95b2-df12c0853cd6" Nov 01 06:30:04.622155 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.622131 5040 generic.go:356] "Generic (PLEG): container finished" podID="1575b2f5-afcc-4d4e-8db0-98e419fced3f" containerID="0df498437e0b0b598a72e861d5c1ebe3ec45deb362bf5e5e3a30285c7c8551bd" exitCode=60 Nov 01 06:30:04.622240 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.622209 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891910-pt7jg" event={"ID":"1575b2f5-afcc-4d4e-8db0-98e419fced3f","Type":"ContainerDied","Data":"0df498437e0b0b598a72e861d5c1ebe3ec45deb362bf5e5e3a30285c7c8551bd"} Nov 01 06:30:04.622434 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.622417 5040 scope.go:122] "RemoveContainer" containerID="0df498437e0b0b598a72e861d5c1ebe3ec45deb362bf5e5e3a30285c7c8551bd" Nov 01 06:30:04.622728 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:30:04.622710 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891910-pt7jg_odh-ai-gateway-infra(1575b2f5-afcc-4d4e-8db0-98e419fced3f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891910-pt7jg" podUID="1575b2f5-afcc-4d4e-8db0-98e419fced3f" Nov 01 06:30:04.624216 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.624198 5040 generic.go:356] "Generic (PLEG): container finished" podID="98183e7b-70c1-4af0-9b64-9b4cba9424a1" containerID="39612278cd707ab2a21a2010ed477573cee16a1f8ba41307c27ac5dc15d86eb1" exitCode=60 Nov 01 06:30:04.624295 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.624277 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891910-6ptzx" event={"ID":"98183e7b-70c1-4af0-9b64-9b4cba9424a1","Type":"ContainerDied","Data":"39612278cd707ab2a21a2010ed477573cee16a1f8ba41307c27ac5dc15d86eb1"} Nov 01 06:30:04.624446 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.624433 5040 scope.go:122] "RemoveContainer" containerID="39612278cd707ab2a21a2010ed477573cee16a1f8ba41307c27ac5dc15d86eb1" Nov 01 06:30:04.624692 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:30:04.624673 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7d25db-29891910-6ptzx_odh-ai-gateway-infra(98183e7b-70c1-4af0-9b64-9b4cba9424a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891910-6ptzx" podUID="98183e7b-70c1-4af0-9b64-9b4cba9424a1" Nov 01 06:30:04.626147 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.626128 5040 generic.go:356] "Generic (PLEG): container finished" podID="66ba8fe2-b2da-4281-aec7-7f2b8440e548" containerID="990141bde89912ea666dd6c6d989fb5c9816bb256ee4aafb49df44e89f14618c" exitCode=60 Nov 01 06:30:04.626238 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.626207 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891910-cn5cf" event={"ID":"66ba8fe2-b2da-4281-aec7-7f2b8440e548","Type":"ContainerDied","Data":"990141bde89912ea666dd6c6d989fb5c9816bb256ee4aafb49df44e89f14618c"} Nov 01 06:30:04.626435 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.626419 5040 scope.go:122] "RemoveContainer" containerID="990141bde89912ea666dd6c6d989fb5c9816bb256ee4aafb49df44e89f14618c" Nov 01 06:30:04.626700 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:30:04.626680 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891910-cn5cf_odh-ai-gateway-infra(66ba8fe2-b2da-4281-aec7-7f2b8440e548)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891910-cn5cf" podUID="66ba8fe2-b2da-4281-aec7-7f2b8440e548" Nov 01 06:30:04.633882 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.633858 5040 scope.go:122] "RemoveContainer" containerID="10d10320f776c0a66e97594767d20bfd1174678f2cd2854f6ad18a2c3bec3825" Nov 01 06:30:04.646889 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.646867 5040 scope.go:122] "RemoveContainer" containerID="c1e1ba7da29e26419fa4675849bd4e8b52a4601cdad830ec7b214772bdd85366" Nov 01 06:30:04.654873 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.654836 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891910-6ngcr" podStartSLOduration=4.654820901 podStartE2EDuration="4.654820901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:04.651284647 +0000 UTC m=+2683.306401814" watchObservedRunningTime="2026-11-01 06:30:04.654820901 +0000 UTC m=+2683.309938069" Nov 01 06:30:04.660134 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.660118 5040 scope.go:122] "RemoveContainer" containerID="56ccee39e001011b84e90fa72a770f19db0f3519451bd23644b45354ccde32dd" Nov 01 06:30:04.673531 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.673472 5040 scope.go:122] "RemoveContainer" containerID="14502ae2a339def2956a977bf5d60bc4cb0b29ac992cab36ac60568429bfaeb5" Nov 01 06:30:04.686333 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.686300 5040 scope.go:122] "RemoveContainer" containerID="98cd44a7defe0e020859b1f92c625185f1eb4cfffcdb6156ecf61509ce935f27" Nov 01 06:30:04.724952 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.724929 5040 scope.go:122] "RemoveContainer" containerID="93fb7a4b3ae421cf7ef6a428c86c5709f800893b481500283d15d401d743af5e" Nov 01 06:30:04.742797 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.742777 5040 scope.go:122] "RemoveContainer" containerID="8ca6788518aac4758cfe552a22fa3d4d04eff8852c61485a06e1b6bf14edfea9" Nov 01 06:30:04.756890 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.756871 5040 scope.go:122] "RemoveContainer" containerID="4b8f77324e596a2dc3958a77c2b213556e62800aa37f05ac279239010a42e428" Nov 01 06:30:04.783724 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:04.783677 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891910-kbwkk" podStartSLOduration=4.783663349 podStartE2EDuration="4.783663349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:04.782575253 +0000 UTC m=+2683.437692421" watchObservedRunningTime="2026-11-01 06:30:04.783663349 +0000 UTC m=+2683.438780572" Nov 01 06:30:05.036258 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:05.036189 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891910-cn5cf" podStartSLOduration=5.036172998 podStartE2EDuration="5.036172998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:04.918856281 +0000 UTC m=+2683.573973451" watchObservedRunningTime="2026-11-01 06:30:05.036172998 +0000 UTC m=+2683.691290162" Nov 01 06:30:05.635706 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:05.635678 5040 scope.go:122] "RemoveContainer" containerID="8076a746a19b6ea48ae69f8b6cad719b0d910f0a0ae64ab20eab477efc917449" Nov 01 06:30:05.636158 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:30:05.636004 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-3986bf-29891910-6ngcr_odh-ai-gateway-infra(15896689-4508-43e1-95b2-df12c0853cd6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891910-6ngcr" podUID="15896689-4508-43e1-95b2-df12c0853cd6" Nov 01 06:30:05.637374 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:05.637349 5040 scope.go:122] "RemoveContainer" containerID="0df498437e0b0b598a72e861d5c1ebe3ec45deb362bf5e5e3a30285c7c8551bd" Nov 01 06:30:05.637641 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:30:05.637621 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891910-pt7jg_odh-ai-gateway-infra(1575b2f5-afcc-4d4e-8db0-98e419fced3f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891910-pt7jg" podUID="1575b2f5-afcc-4d4e-8db0-98e419fced3f" Nov 01 06:30:05.639197 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:05.639180 5040 scope.go:122] "RemoveContainer" containerID="39612278cd707ab2a21a2010ed477573cee16a1f8ba41307c27ac5dc15d86eb1" Nov 01 06:30:05.639404 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:30:05.639384 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7d25db-29891910-6ptzx_odh-ai-gateway-infra(98183e7b-70c1-4af0-9b64-9b4cba9424a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891910-6ptzx" podUID="98183e7b-70c1-4af0-9b64-9b4cba9424a1" Nov 01 06:30:05.640808 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:05.640791 5040 scope.go:122] "RemoveContainer" containerID="990141bde89912ea666dd6c6d989fb5c9816bb256ee4aafb49df44e89f14618c" Nov 01 06:30:05.641034 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:30:05.641012 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891910-cn5cf_odh-ai-gateway-infra(66ba8fe2-b2da-4281-aec7-7f2b8440e548)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891910-cn5cf" podUID="66ba8fe2-b2da-4281-aec7-7f2b8440e548" Nov 01 06:30:05.642654 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:05.642639 5040 scope.go:122] "RemoveContainer" containerID="3982ea529b948eeb5e1da63a5870cdd186b028fd73ed65c3e5b1e18a60d7d403" Nov 01 06:30:05.642836 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:30:05.642816 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891910-4x68f_odh-ai-gateway-infra(1da58795-e0ae-4db3-9774-0c8d997352a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891910-4x68f" podUID="1da58795-e0ae-4db3-9774-0c8d997352a6" Nov 01 06:30:05.643919 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:05.643901 5040 scope.go:122] "RemoveContainer" containerID="8182279e7a331feb0fd1f9194a69a35536d4f72c76c84748b061f6ba5e0990a5" Nov 01 06:30:05.644082 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:30:05.644062 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-f73e89-29891910-2vgzz_odh-ai-gateway-infra(dd27d740-cb57-479b-97ca-517a09d689b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891910-2vgzz" podUID="dd27d740-cb57-479b-97ca-517a09d689b3" Nov 01 06:30:05.645457 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:05.645441 5040 scope.go:122] "RemoveContainer" containerID="8017e9c7e75da8416033457984dec78d13afe6e9ff7ffd6df112464305d16b1b" Nov 01 06:30:05.645629 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:30:05.645611 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-745a18-29891910-gw4vg_odh-ai-gateway-infra(aecbbf55-6343-4752-8e2b-0f3771c4df57)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-745a18-29891910-gw4vg" podUID="aecbbf55-6343-4752-8e2b-0f3771c4df57" Nov 01 06:30:05.646939 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:05.646924 5040 scope.go:122] "RemoveContainer" containerID="ea3f072afdebdedf1a5d87af92c1c185d5df8b08a114f37a2b410f473d9e0d39" Nov 01 06:30:05.647137 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:30:05.647115 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891910-g2s6x_odh-ai-gateway-infra(7c7c88e9-2475-4bf5-9c1a-426c1643312c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891910-g2s6x" podUID="7c7c88e9-2475-4bf5-9c1a-426c1643312c" Nov 01 06:30:05.648438 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:05.648420 5040 scope.go:122] "RemoveContainer" containerID="9844b6f27212b2bc32183218acd0c59ae1eae92c5643b57a19551a4e5086af13" Nov 01 06:30:05.648648 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:30:05.648627 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891910-cmplg_odh-ai-gateway-infra(4218688b-c54f-47cc-a11f-8fe7dde5cc48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-cmplg" podUID="4218688b-c54f-47cc-a11f-8fe7dde5cc48" Nov 01 06:30:05.649896 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:05.649880 5040 scope.go:122] "RemoveContainer" containerID="2182bd2fd02855b1175c942c5ccf3640415d0bb9ab0eb0f7af8c0c47327495e7" Nov 01 06:30:05.650044 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:30:05.650028 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891910-kbwkk_odh-ai-gateway-infra(a282a96e-d0fc-4d97-b2e1-35b8c593bcfc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891910-kbwkk" podUID="a282a96e-d0fc-4d97-b2e1-35b8c593bcfc" Nov 01 06:30:09.978065 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:30:09.978024 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:30:12.703375 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:12.703279 5040 generic.go:356] "Generic (PLEG): container finished" podID="1978f64d-b811-47bf-862a-aacbde03b6e9" containerID="d007250bc3884dd71ace59931222b603f24b26e9f439455797992cd76c981cc4" exitCode=0 Nov 01 06:30:12.703375 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:12.703360 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" event={"ID":"1978f64d-b811-47bf-862a-aacbde03b6e9","Type":"ContainerDied","Data":"d007250bc3884dd71ace59931222b603f24b26e9f439455797992cd76c981cc4"} Nov 01 06:30:12.703846 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:12.703816 5040 scope.go:122] "RemoveContainer" containerID="d007250bc3884dd71ace59931222b603f24b26e9f439455797992cd76c981cc4" Nov 01 06:30:12.705315 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:12.705285 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:30:12.705379 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:12.705370 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:30:12.705424 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:12.705412 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:30:13.048977 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:13.048941 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a355f03b-qsd22"] Nov 01 06:30:13.052765 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:13.052738 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a355f03b-qsd22"] Nov 01 06:30:13.056131 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:13.056101 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3f698a5-tcn6w"] Nov 01 06:30:13.059273 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:13.059247 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3f698a5-tcn6w"] Nov 01 06:30:13.526802 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:13.526769 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="799dab13-f58a-4400-ab1f-39a0b86d7e24" path="/var/lib/kubelet/pods/799dab13-f58a-4400-ab1f-39a0b86d7e24/volumes" Nov 01 06:30:13.527255 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:13.527229 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0526dc9-6d93-44f5-8f41-094310810816" path="/var/lib/kubelet/pods/a0526dc9-6d93-44f5-8f41-094310810816/volumes" Nov 01 06:30:13.725422 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:13.725382 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" event={"ID":"1978f64d-b811-47bf-862a-aacbde03b6e9","Type":"ContainerStarted","Data":"390ee69e7c4e7e5b2acba384f2aa0c5cbd6dba9aa60a046b0bf23504347080a7"} Nov 01 06:30:14.515504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:14.514753 5040 scope.go:122] "RemoveContainer" containerID="ea3f072afdebdedf1a5d87af92c1c185d5df8b08a114f37a2b410f473d9e0d39" Nov 01 06:30:14.515504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:14.515495 5040 scope.go:122] "RemoveContainer" containerID="2182bd2fd02855b1175c942c5ccf3640415d0bb9ab0eb0f7af8c0c47327495e7" Nov 01 06:30:14.689893 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:14.689857 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:30:14.690929 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:14.690904 5040 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 01 06:30:14.691004 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:14.690970 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" podUID="1978f64d-b811-47bf-862a-aacbde03b6e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 01 06:30:15.690032 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:15.689992 5040 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 01 06:30:15.690425 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:15.690050 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" podUID="1978f64d-b811-47bf-862a-aacbde03b6e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 01 06:30:15.743958 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:15.743924 5040 generic.go:356] "Generic (PLEG): container finished" podID="7c7c88e9-2475-4bf5-9c1a-426c1643312c" containerID="7f52243bac16145d618dead780759673afccafdcbe1be94ba3cfcc1434ccef37" exitCode=60 Nov 01 06:30:15.744147 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:15.744005 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891910-g2s6x" event={"ID":"7c7c88e9-2475-4bf5-9c1a-426c1643312c","Type":"ContainerDied","Data":"7f52243bac16145d618dead780759673afccafdcbe1be94ba3cfcc1434ccef37"} Nov 01 06:30:15.744147 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:15.744076 5040 scope.go:122] "RemoveContainer" containerID="ea3f072afdebdedf1a5d87af92c1c185d5df8b08a114f37a2b410f473d9e0d39" Nov 01 06:30:15.744436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:15.744416 5040 scope.go:122] "RemoveContainer" containerID="7f52243bac16145d618dead780759673afccafdcbe1be94ba3cfcc1434ccef37" Nov 01 06:30:15.744896 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:30:15.744861 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891910-g2s6x_odh-ai-gateway-infra(7c7c88e9-2475-4bf5-9c1a-426c1643312c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891910-g2s6x" podUID="7c7c88e9-2475-4bf5-9c1a-426c1643312c" Nov 01 06:30:15.746202 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:15.746181 5040 generic.go:356] "Generic (PLEG): container finished" podID="a282a96e-d0fc-4d97-b2e1-35b8c593bcfc" containerID="308ae79f33818c9decd9a7e8d0b22d29dd2cc12be2ba633c0ac43d50b70612f1" exitCode=60 Nov 01 06:30:15.746284 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:15.746253 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891910-kbwkk" event={"ID":"a282a96e-d0fc-4d97-b2e1-35b8c593bcfc","Type":"ContainerDied","Data":"308ae79f33818c9decd9a7e8d0b22d29dd2cc12be2ba633c0ac43d50b70612f1"} Nov 01 06:30:15.746628 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:15.746613 5040 scope.go:122] "RemoveContainer" containerID="308ae79f33818c9decd9a7e8d0b22d29dd2cc12be2ba633c0ac43d50b70612f1" Nov 01 06:30:15.746847 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:30:15.746829 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891910-kbwkk_odh-ai-gateway-infra(a282a96e-d0fc-4d97-b2e1-35b8c593bcfc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891910-kbwkk" podUID="a282a96e-d0fc-4d97-b2e1-35b8c593bcfc" Nov 01 06:30:15.757265 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:15.757238 5040 scope.go:122] "RemoveContainer" containerID="2182bd2fd02855b1175c942c5ccf3640415d0bb9ab0eb0f7af8c0c47327495e7" Nov 01 06:30:15.825205 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:15.825177 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:30:16.518517 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:16.512770 5040 scope.go:122] "RemoveContainer" containerID="8182279e7a331feb0fd1f9194a69a35536d4f72c76c84748b061f6ba5e0990a5" Nov 01 06:30:16.518517 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:16.514250 5040 scope.go:122] "RemoveContainer" containerID="39612278cd707ab2a21a2010ed477573cee16a1f8ba41307c27ac5dc15d86eb1" Nov 01 06:30:16.518517 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:16.515351 5040 scope.go:122] "RemoveContainer" containerID="8076a746a19b6ea48ae69f8b6cad719b0d910f0a0ae64ab20eab477efc917449" Nov 01 06:30:16.689853 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:16.689817 5040 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 01 06:30:16.690036 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:16.689869 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" podUID="1978f64d-b811-47bf-862a-aacbde03b6e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 01 06:30:16.891081 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:16.891040 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891910-g2s6x"] Nov 01 06:30:17.032464 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.031889 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891910-kbwkk"] Nov 01 06:30:17.172505 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.169648 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891910-g2s6x" Nov 01 06:30:17.255748 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.255710 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c7c88e9-2475-4bf5-9c1a-426c1643312c-kube-api-access-dfxkb\" (UniqueName: \"kubernetes.io/projected/7c7c88e9-2475-4bf5-9c1a-426c1643312c-kube-api-access-dfxkb\") pod \"7c7c88e9-2475-4bf5-9c1a-426c1643312c\" (UID: \"7c7c88e9-2475-4bf5-9c1a-426c1643312c\") " Nov 01 06:30:17.255954 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.255832 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c7c88e9-2475-4bf5-9c1a-426c1643312c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7c88e9-2475-4bf5-9c1a-426c1643312c-maas-api-service-ca\") pod \"7c7c88e9-2475-4bf5-9c1a-426c1643312c\" (UID: \"7c7c88e9-2475-4bf5-9c1a-426c1643312c\") " Nov 01 06:30:17.256559 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.256532 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c7c88e9-2475-4bf5-9c1a-426c1643312c-maas-api-service-ca" pod "7c7c88e9-2475-4bf5-9c1a-426c1643312c" (UID: "7c7c88e9-2475-4bf5-9c1a-426c1643312c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:30:17.259341 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.259314 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c7c88e9-2475-4bf5-9c1a-426c1643312c-kube-api-access-dfxkb" pod "7c7c88e9-2475-4bf5-9c1a-426c1643312c" (UID: "7c7c88e9-2475-4bf5-9c1a-426c1643312c"). InnerVolumeSpecName "kube-api-access-dfxkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:30:17.341197 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.341128 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891910-kbwkk" Nov 01 06:30:17.357276 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.357244 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a282a96e-d0fc-4d97-b2e1-35b8c593bcfc-kube-api-access-nrwds\" (UniqueName: \"kubernetes.io/projected/a282a96e-d0fc-4d97-b2e1-35b8c593bcfc-kube-api-access-nrwds\") pod \"a282a96e-d0fc-4d97-b2e1-35b8c593bcfc\" (UID: \"a282a96e-d0fc-4d97-b2e1-35b8c593bcfc\") " Nov 01 06:30:17.357471 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.357450 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a282a96e-d0fc-4d97-b2e1-35b8c593bcfc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a282a96e-d0fc-4d97-b2e1-35b8c593bcfc-maas-api-service-ca\") pod \"a282a96e-d0fc-4d97-b2e1-35b8c593bcfc\" (UID: \"a282a96e-d0fc-4d97-b2e1-35b8c593bcfc\") " Nov 01 06:30:17.357821 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.357796 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a282a96e-d0fc-4d97-b2e1-35b8c593bcfc-maas-api-service-ca" pod "a282a96e-d0fc-4d97-b2e1-35b8c593bcfc" (UID: "a282a96e-d0fc-4d97-b2e1-35b8c593bcfc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:30:17.357904 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.357852 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7c88e9-2475-4bf5-9c1a-426c1643312c-maas-api-service-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:17.357904 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.357868 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfxkb\" (UniqueName: \"kubernetes.io/projected/7c7c88e9-2475-4bf5-9c1a-426c1643312c-kube-api-access-dfxkb\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:17.359447 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.359425 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a282a96e-d0fc-4d97-b2e1-35b8c593bcfc-kube-api-access-nrwds" pod "a282a96e-d0fc-4d97-b2e1-35b8c593bcfc" (UID: "a282a96e-d0fc-4d97-b2e1-35b8c593bcfc"). InnerVolumeSpecName "kube-api-access-nrwds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:30:17.458557 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.458523 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a282a96e-d0fc-4d97-b2e1-35b8c593bcfc-maas-api-service-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:17.458557 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.458549 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrwds\" (UniqueName: \"kubernetes.io/projected/a282a96e-d0fc-4d97-b2e1-35b8c593bcfc-kube-api-access-nrwds\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:17.522408 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.522372 5040 scope.go:122] "RemoveContainer" containerID="0df498437e0b0b598a72e861d5c1ebe3ec45deb362bf5e5e3a30285c7c8551bd" Nov 01 06:30:17.523649 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.523626 5040 scope.go:122] "RemoveContainer" containerID="8017e9c7e75da8416033457984dec78d13afe6e9ff7ffd6df112464305d16b1b" Nov 01 06:30:17.526838 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.525143 5040 scope.go:122] "RemoveContainer" containerID="9844b6f27212b2bc32183218acd0c59ae1eae92c5643b57a19551a4e5086af13" Nov 01 06:30:17.690106 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.690068 5040 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 01 06:30:17.690272 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.690122 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" podUID="1978f64d-b811-47bf-862a-aacbde03b6e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 01 06:30:17.768112 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.768070 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891910-kbwkk" event={"ID":"a282a96e-d0fc-4d97-b2e1-35b8c593bcfc","Type":"ContainerDied","Data":"de53d7e9af4308402b9f8378d58f17ea28be1d6c42b8e4e1bcfd1d80ae6e854d"} Nov 01 06:30:17.768112 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.768098 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891910-kbwkk" Nov 01 06:30:17.768385 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.768146 5040 scope.go:122] "RemoveContainer" containerID="308ae79f33818c9decd9a7e8d0b22d29dd2cc12be2ba633c0ac43d50b70612f1" Nov 01 06:30:17.770868 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.770836 5040 generic.go:356] "Generic (PLEG): container finished" podID="15896689-4508-43e1-95b2-df12c0853cd6" containerID="6bf777b11c84317c05493c255f2d6f0c9249dc5f7b7c453dd87cb3b2260844ff" exitCode=60 Nov 01 06:30:17.770990 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.770907 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891910-6ngcr" event={"ID":"15896689-4508-43e1-95b2-df12c0853cd6","Type":"ContainerDied","Data":"6bf777b11c84317c05493c255f2d6f0c9249dc5f7b7c453dd87cb3b2260844ff"} Nov 01 06:30:17.771362 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.771344 5040 scope.go:122] "RemoveContainer" containerID="6bf777b11c84317c05493c255f2d6f0c9249dc5f7b7c453dd87cb3b2260844ff" Nov 01 06:30:17.771725 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:30:17.771699 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-3986bf-29891910-6ngcr_odh-ai-gateway-infra(15896689-4508-43e1-95b2-df12c0853cd6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891910-6ngcr" podUID="15896689-4508-43e1-95b2-df12c0853cd6" Nov 01 06:30:17.773857 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.773669 5040 generic.go:356] "Generic (PLEG): container finished" podID="98183e7b-70c1-4af0-9b64-9b4cba9424a1" containerID="38546475b521facbf648c215c85a39c30f5730bd9bc2c9167bd6caa572fa4ffe" exitCode=60 Nov 01 06:30:17.773857 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.773735 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891910-6ptzx" event={"ID":"98183e7b-70c1-4af0-9b64-9b4cba9424a1","Type":"ContainerDied","Data":"38546475b521facbf648c215c85a39c30f5730bd9bc2c9167bd6caa572fa4ffe"} Nov 01 06:30:17.774132 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.774112 5040 scope.go:122] "RemoveContainer" containerID="38546475b521facbf648c215c85a39c30f5730bd9bc2c9167bd6caa572fa4ffe" Nov 01 06:30:17.774376 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:30:17.774357 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7d25db-29891910-6ptzx_odh-ai-gateway-infra(98183e7b-70c1-4af0-9b64-9b4cba9424a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891910-6ptzx" podUID="98183e7b-70c1-4af0-9b64-9b4cba9424a1" Nov 01 06:30:17.776569 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.776547 5040 generic.go:356] "Generic (PLEG): container finished" podID="dd27d740-cb57-479b-97ca-517a09d689b3" containerID="0b80ef28ab9d14363594fc8cbcc63950701cd04a5b05cb7b4190811deabc2dc5" exitCode=60 Nov 01 06:30:17.776661 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.776627 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891910-2vgzz" event={"ID":"dd27d740-cb57-479b-97ca-517a09d689b3","Type":"ContainerDied","Data":"0b80ef28ab9d14363594fc8cbcc63950701cd04a5b05cb7b4190811deabc2dc5"} Nov 01 06:30:17.777081 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.777029 5040 scope.go:122] "RemoveContainer" containerID="0b80ef28ab9d14363594fc8cbcc63950701cd04a5b05cb7b4190811deabc2dc5" Nov 01 06:30:17.777450 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:30:17.777298 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-f73e89-29891910-2vgzz_odh-ai-gateway-infra(dd27d740-cb57-479b-97ca-517a09d689b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891910-2vgzz" podUID="dd27d740-cb57-479b-97ca-517a09d689b3" Nov 01 06:30:17.778416 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.778384 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891910-g2s6x" event={"ID":"7c7c88e9-2475-4bf5-9c1a-426c1643312c","Type":"ContainerDied","Data":"c6ab5b64e3aae7b6175da312733ac6ba8a60228839cbb05bb4ee0e6b66ceabc5"} Nov 01 06:30:17.778513 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.778417 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891910-g2s6x" Nov 01 06:30:17.786880 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.786856 5040 scope.go:122] "RemoveContainer" containerID="8076a746a19b6ea48ae69f8b6cad719b0d910f0a0ae64ab20eab477efc917449" Nov 01 06:30:17.829206 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.829178 5040 scope.go:122] "RemoveContainer" containerID="39612278cd707ab2a21a2010ed477573cee16a1f8ba41307c27ac5dc15d86eb1" Nov 01 06:30:17.844278 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.844256 5040 scope.go:122] "RemoveContainer" containerID="8182279e7a331feb0fd1f9194a69a35536d4f72c76c84748b061f6ba5e0990a5" Nov 01 06:30:17.859818 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:17.859796 5040 scope.go:122] "RemoveContainer" containerID="7f52243bac16145d618dead780759673afccafdcbe1be94ba3cfcc1434ccef37" Nov 01 06:30:18.187868 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:18.187827 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891910-kbwkk"] Nov 01 06:30:18.224102 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:18.224058 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891910-kbwkk"] Nov 01 06:30:18.334508 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:18.334449 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891910-g2s6x"] Nov 01 06:30:18.345535 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:18.343874 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891910-g2s6x"] Nov 01 06:30:18.511337 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:18.507740 5040 scope.go:122] "RemoveContainer" containerID="990141bde89912ea666dd6c6d989fb5c9816bb256ee4aafb49df44e89f14618c" Nov 01 06:30:18.511337 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:18.507944 5040 scope.go:122] "RemoveContainer" containerID="3982ea529b948eeb5e1da63a5870cdd186b028fd73ed65c3e5b1e18a60d7d403" Nov 01 06:30:18.690127 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:18.690093 5040 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 01 06:30:18.690328 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:18.690147 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" podUID="1978f64d-b811-47bf-862a-aacbde03b6e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 01 06:30:18.791196 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:18.791094 5040 generic.go:356] "Generic (PLEG): container finished" podID="aecbbf55-6343-4752-8e2b-0f3771c4df57" containerID="5e88d34db60185a8e3b5b76d768ac5a0214145212d713c2a16066c5a9382c69a" exitCode=60 Nov 01 06:30:18.791359 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:18.791184 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-745a18-29891910-gw4vg" event={"ID":"aecbbf55-6343-4752-8e2b-0f3771c4df57","Type":"ContainerDied","Data":"5e88d34db60185a8e3b5b76d768ac5a0214145212d713c2a16066c5a9382c69a"} Nov 01 06:30:18.791359 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:18.791264 5040 scope.go:122] "RemoveContainer" containerID="8017e9c7e75da8416033457984dec78d13afe6e9ff7ffd6df112464305d16b1b" Nov 01 06:30:18.792291 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:18.791774 5040 scope.go:122] "RemoveContainer" containerID="5e88d34db60185a8e3b5b76d768ac5a0214145212d713c2a16066c5a9382c69a" Nov 01 06:30:18.792291 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:30:18.792085 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-745a18-29891910-gw4vg_odh-ai-gateway-infra(aecbbf55-6343-4752-8e2b-0f3771c4df57)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-745a18-29891910-gw4vg" podUID="aecbbf55-6343-4752-8e2b-0f3771c4df57" Nov 01 06:30:18.814335 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:18.814272 5040 generic.go:356] "Generic (PLEG): container finished" podID="4218688b-c54f-47cc-a11f-8fe7dde5cc48" containerID="cae240eef2073b3501cb94754f84e55c1dbc88696ba254f01fccb9948b67e45e" exitCode=60 Nov 01 06:30:18.814559 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:18.814424 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-cmplg" event={"ID":"4218688b-c54f-47cc-a11f-8fe7dde5cc48","Type":"ContainerDied","Data":"cae240eef2073b3501cb94754f84e55c1dbc88696ba254f01fccb9948b67e45e"} Nov 01 06:30:18.818512 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:18.814945 5040 scope.go:122] "RemoveContainer" containerID="cae240eef2073b3501cb94754f84e55c1dbc88696ba254f01fccb9948b67e45e" Nov 01 06:30:18.818512 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:30:18.815286 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891910-cmplg_odh-ai-gateway-infra(4218688b-c54f-47cc-a11f-8fe7dde5cc48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-cmplg" podUID="4218688b-c54f-47cc-a11f-8fe7dde5cc48" Nov 01 06:30:18.822872 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:18.821432 5040 generic.go:356] "Generic (PLEG): container finished" podID="1575b2f5-afcc-4d4e-8db0-98e419fced3f" containerID="86d0b25dcfbc7775d23d802c985e77c9df389dcdbb1f18b5bc1a682b1348d902" exitCode=60 Nov 01 06:30:18.822872 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:18.821576 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891910-pt7jg" event={"ID":"1575b2f5-afcc-4d4e-8db0-98e419fced3f","Type":"ContainerDied","Data":"86d0b25dcfbc7775d23d802c985e77c9df389dcdbb1f18b5bc1a682b1348d902"} Nov 01 06:30:18.822872 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:18.821998 5040 scope.go:122] "RemoveContainer" containerID="86d0b25dcfbc7775d23d802c985e77c9df389dcdbb1f18b5bc1a682b1348d902" Nov 01 06:30:18.822872 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:30:18.822316 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891910-pt7jg_odh-ai-gateway-infra(1575b2f5-afcc-4d4e-8db0-98e419fced3f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891910-pt7jg" podUID="1575b2f5-afcc-4d4e-8db0-98e419fced3f" Nov 01 06:30:18.836995 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:18.836971 5040 scope.go:122] "RemoveContainer" containerID="9844b6f27212b2bc32183218acd0c59ae1eae92c5643b57a19551a4e5086af13" Nov 01 06:30:18.851724 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:18.851699 5040 scope.go:122] "RemoveContainer" containerID="0df498437e0b0b598a72e861d5c1ebe3ec45deb362bf5e5e3a30285c7c8551bd" Nov 01 06:30:18.953844 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:18.953810 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891910-6ptzx"] Nov 01 06:30:19.117021 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.116988 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891910-6ngcr"] Nov 01 06:30:19.142516 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.142133 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891910-6ptzx" Nov 01 06:30:19.177752 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.177723 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98183e7b-70c1-4af0-9b64-9b4cba9424a1-kube-api-access-rbg2c\" (UniqueName: \"kubernetes.io/projected/98183e7b-70c1-4af0-9b64-9b4cba9424a1-kube-api-access-rbg2c\") pod \"98183e7b-70c1-4af0-9b64-9b4cba9424a1\" (UID: \"98183e7b-70c1-4af0-9b64-9b4cba9424a1\") " Nov 01 06:30:19.177927 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.177889 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98183e7b-70c1-4af0-9b64-9b4cba9424a1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98183e7b-70c1-4af0-9b64-9b4cba9424a1-maas-api-service-ca\") pod \"98183e7b-70c1-4af0-9b64-9b4cba9424a1\" (UID: \"98183e7b-70c1-4af0-9b64-9b4cba9424a1\") " Nov 01 06:30:19.178268 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.178236 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98183e7b-70c1-4af0-9b64-9b4cba9424a1-maas-api-service-ca" pod "98183e7b-70c1-4af0-9b64-9b4cba9424a1" (UID: "98183e7b-70c1-4af0-9b64-9b4cba9424a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:30:19.179939 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.179907 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98183e7b-70c1-4af0-9b64-9b4cba9424a1-kube-api-access-rbg2c" pod "98183e7b-70c1-4af0-9b64-9b4cba9424a1" (UID: "98183e7b-70c1-4af0-9b64-9b4cba9424a1"). InnerVolumeSpecName "kube-api-access-rbg2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:30:19.280319 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.280257 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbg2c\" (UniqueName: \"kubernetes.io/projected/98183e7b-70c1-4af0-9b64-9b4cba9424a1-kube-api-access-rbg2c\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:19.280319 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.280293 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98183e7b-70c1-4af0-9b64-9b4cba9424a1-maas-api-service-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:19.300833 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.300799 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891910-6ngcr" Nov 01 06:30:19.381192 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.381096 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15896689-4508-43e1-95b2-df12c0853cd6-kube-api-access-2n5lz\" (UniqueName: \"kubernetes.io/projected/15896689-4508-43e1-95b2-df12c0853cd6-kube-api-access-2n5lz\") pod \"15896689-4508-43e1-95b2-df12c0853cd6\" (UID: \"15896689-4508-43e1-95b2-df12c0853cd6\") " Nov 01 06:30:19.381362 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.381194 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15896689-4508-43e1-95b2-df12c0853cd6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15896689-4508-43e1-95b2-df12c0853cd6-maas-api-service-ca\") pod \"15896689-4508-43e1-95b2-df12c0853cd6\" (UID: \"15896689-4508-43e1-95b2-df12c0853cd6\") " Nov 01 06:30:19.381612 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.381584 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15896689-4508-43e1-95b2-df12c0853cd6-maas-api-service-ca" pod "15896689-4508-43e1-95b2-df12c0853cd6" (UID: "15896689-4508-43e1-95b2-df12c0853cd6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:30:19.383358 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.383338 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15896689-4508-43e1-95b2-df12c0853cd6-kube-api-access-2n5lz" pod "15896689-4508-43e1-95b2-df12c0853cd6" (UID: "15896689-4508-43e1-95b2-df12c0853cd6"). InnerVolumeSpecName "kube-api-access-2n5lz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:30:19.468437 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.468403 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891910-2vgzz"] Nov 01 06:30:19.482469 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.482439 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15896689-4508-43e1-95b2-df12c0853cd6-maas-api-service-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:19.482469 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.482466 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n5lz\" (UniqueName: \"kubernetes.io/projected/15896689-4508-43e1-95b2-df12c0853cd6-kube-api-access-2n5lz\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:19.517159 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.517126 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c7c88e9-2475-4bf5-9c1a-426c1643312c" path="/var/lib/kubelet/pods/7c7c88e9-2475-4bf5-9c1a-426c1643312c/volumes" Nov 01 06:30:19.517705 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.517684 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a282a96e-d0fc-4d97-b2e1-35b8c593bcfc" path="/var/lib/kubelet/pods/a282a96e-d0fc-4d97-b2e1-35b8c593bcfc/volumes" Nov 01 06:30:19.651461 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.651395 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891910-2vgzz" Nov 01 06:30:19.684850 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.684807 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd27d740-cb57-479b-97ca-517a09d689b3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd27d740-cb57-479b-97ca-517a09d689b3-maas-api-service-ca\") pod \"dd27d740-cb57-479b-97ca-517a09d689b3\" (UID: \"dd27d740-cb57-479b-97ca-517a09d689b3\") " Nov 01 06:30:19.685031 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.684987 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd27d740-cb57-479b-97ca-517a09d689b3-kube-api-access-nfhpp\" (UniqueName: \"kubernetes.io/projected/dd27d740-cb57-479b-97ca-517a09d689b3-kube-api-access-nfhpp\") pod \"dd27d740-cb57-479b-97ca-517a09d689b3\" (UID: \"dd27d740-cb57-479b-97ca-517a09d689b3\") " Nov 01 06:30:19.685180 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.685156 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd27d740-cb57-479b-97ca-517a09d689b3-maas-api-service-ca" pod "dd27d740-cb57-479b-97ca-517a09d689b3" (UID: "dd27d740-cb57-479b-97ca-517a09d689b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:30:19.685353 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.685337 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd27d740-cb57-479b-97ca-517a09d689b3-maas-api-service-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:19.687047 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.687024 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd27d740-cb57-479b-97ca-517a09d689b3-kube-api-access-nfhpp" pod "dd27d740-cb57-479b-97ca-517a09d689b3" (UID: "dd27d740-cb57-479b-97ca-517a09d689b3"). InnerVolumeSpecName "kube-api-access-nfhpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:30:19.693854 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.693831 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:30:19.694867 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.694849 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:30:19.787222 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.787166 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfhpp\" (UniqueName: \"kubernetes.io/projected/dd27d740-cb57-479b-97ca-517a09d689b3-kube-api-access-nfhpp\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:19.835957 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.835922 5040 generic.go:356] "Generic (PLEG): container finished" podID="1da58795-e0ae-4db3-9774-0c8d997352a6" containerID="45542b10ecac91670fe3f2b7f347376c0dab918bff29207b7d77073ae3386934" exitCode=60 Nov 01 06:30:19.836131 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.835998 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891910-4x68f" event={"ID":"1da58795-e0ae-4db3-9774-0c8d997352a6","Type":"ContainerDied","Data":"45542b10ecac91670fe3f2b7f347376c0dab918bff29207b7d77073ae3386934"} Nov 01 06:30:19.836131 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.836062 5040 scope.go:122] "RemoveContainer" containerID="3982ea529b948eeb5e1da63a5870cdd186b028fd73ed65c3e5b1e18a60d7d403" Nov 01 06:30:19.836508 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.836458 5040 scope.go:122] "RemoveContainer" containerID="45542b10ecac91670fe3f2b7f347376c0dab918bff29207b7d77073ae3386934" Nov 01 06:30:19.836833 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:30:19.836810 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891910-4x68f_odh-ai-gateway-infra(1da58795-e0ae-4db3-9774-0c8d997352a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891910-4x68f" podUID="1da58795-e0ae-4db3-9774-0c8d997352a6" Nov 01 06:30:19.837668 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.837652 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891910-2vgzz" Nov 01 06:30:19.837668 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.837654 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891910-2vgzz" event={"ID":"dd27d740-cb57-479b-97ca-517a09d689b3","Type":"ContainerDied","Data":"c7c2be49bee8858ce7dc14fa3316978c62233e8a145b7dd943fdf7993502a073"} Nov 01 06:30:19.842588 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.842566 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891910-6ngcr" event={"ID":"15896689-4508-43e1-95b2-df12c0853cd6","Type":"ContainerDied","Data":"77d02b08f87b8aa1374b76e41f9cf61e7cbe1993a18546d1865917338f488c44"} Nov 01 06:30:19.842702 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.842607 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891910-6ngcr" Nov 01 06:30:19.846212 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.846196 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891910-6ptzx" Nov 01 06:30:19.846304 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.846230 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891910-6ptzx" event={"ID":"98183e7b-70c1-4af0-9b64-9b4cba9424a1","Type":"ContainerDied","Data":"b61f4c4160e819d3fa44168fb73bd1ef497d63e1b4fa1af516264600e05e05f4"} Nov 01 06:30:19.848740 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.848718 5040 generic.go:356] "Generic (PLEG): container finished" podID="66ba8fe2-b2da-4281-aec7-7f2b8440e548" containerID="67f43db529f5d6d7c093d494fa61a05eeeff29d207e0c9556fb2d481a7d85e9d" exitCode=60 Nov 01 06:30:19.848839 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.848749 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891910-cn5cf" event={"ID":"66ba8fe2-b2da-4281-aec7-7f2b8440e548","Type":"ContainerDied","Data":"67f43db529f5d6d7c093d494fa61a05eeeff29d207e0c9556fb2d481a7d85e9d"} Nov 01 06:30:19.849346 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.849325 5040 scope.go:122] "RemoveContainer" containerID="67f43db529f5d6d7c093d494fa61a05eeeff29d207e0c9556fb2d481a7d85e9d" Nov 01 06:30:19.849685 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:30:19.849661 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891910-cn5cf_odh-ai-gateway-infra(66ba8fe2-b2da-4281-aec7-7f2b8440e548)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891910-cn5cf" podUID="66ba8fe2-b2da-4281-aec7-7f2b8440e548" Nov 01 06:30:19.851574 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.851556 5040 scope.go:122] "RemoveContainer" containerID="0b80ef28ab9d14363594fc8cbcc63950701cd04a5b05cb7b4190811deabc2dc5" Nov 01 06:30:19.860624 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.860608 5040 scope.go:122] "RemoveContainer" containerID="6bf777b11c84317c05493c255f2d6f0c9249dc5f7b7c453dd87cb3b2260844ff" Nov 01 06:30:19.869628 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.869608 5040 scope.go:122] "RemoveContainer" containerID="38546475b521facbf648c215c85a39c30f5730bd9bc2c9167bd6caa572fa4ffe" Nov 01 06:30:19.878387 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.878367 5040 scope.go:122] "RemoveContainer" containerID="990141bde89912ea666dd6c6d989fb5c9816bb256ee4aafb49df44e89f14618c" Nov 01 06:30:19.923128 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:19.923046 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-745a18-29891910-gw4vg"] Nov 01 06:30:20.016398 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.016331 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891910-2vgzz"] Nov 01 06:30:20.021080 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.021050 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891910-2vgzz"] Nov 01 06:30:20.044807 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.044774 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891910-cmplg"] Nov 01 06:30:20.067233 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.067203 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-745a18-29891910-gw4vg" Nov 01 06:30:20.090733 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.090701 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aecbbf55-6343-4752-8e2b-0f3771c4df57-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aecbbf55-6343-4752-8e2b-0f3771c4df57-maas-api-service-ca\") pod \"aecbbf55-6343-4752-8e2b-0f3771c4df57\" (UID: \"aecbbf55-6343-4752-8e2b-0f3771c4df57\") " Nov 01 06:30:20.090930 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.090794 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aecbbf55-6343-4752-8e2b-0f3771c4df57-kube-api-access-lwclz\" (UniqueName: \"kubernetes.io/projected/aecbbf55-6343-4752-8e2b-0f3771c4df57-kube-api-access-lwclz\") pod \"aecbbf55-6343-4752-8e2b-0f3771c4df57\" (UID: \"aecbbf55-6343-4752-8e2b-0f3771c4df57\") " Nov 01 06:30:20.091375 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.091338 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aecbbf55-6343-4752-8e2b-0f3771c4df57-maas-api-service-ca" pod "aecbbf55-6343-4752-8e2b-0f3771c4df57" (UID: "aecbbf55-6343-4752-8e2b-0f3771c4df57"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:30:20.091582 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.091566 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aecbbf55-6343-4752-8e2b-0f3771c4df57-maas-api-service-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:20.092925 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.092901 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aecbbf55-6343-4752-8e2b-0f3771c4df57-kube-api-access-lwclz" pod "aecbbf55-6343-4752-8e2b-0f3771c4df57" (UID: "aecbbf55-6343-4752-8e2b-0f3771c4df57"). InnerVolumeSpecName "kube-api-access-lwclz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:30:20.193168 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.193139 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwclz\" (UniqueName: \"kubernetes.io/projected/aecbbf55-6343-4752-8e2b-0f3771c4df57-kube-api-access-lwclz\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:20.198772 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.198745 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891910-pt7jg"] Nov 01 06:30:20.203315 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.203281 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-cmplg" Nov 01 06:30:20.241574 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.241542 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891910-6ptzx"] Nov 01 06:30:20.245999 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.245974 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891910-6ptzx"] Nov 01 06:30:20.294601 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.294559 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4218688b-c54f-47cc-a11f-8fe7dde5cc48-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4218688b-c54f-47cc-a11f-8fe7dde5cc48-maas-api-service-ca\") pod \"4218688b-c54f-47cc-a11f-8fe7dde5cc48\" (UID: \"4218688b-c54f-47cc-a11f-8fe7dde5cc48\") " Nov 01 06:30:20.295154 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.294680 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4218688b-c54f-47cc-a11f-8fe7dde5cc48-kube-api-access-m45sr\" (UniqueName: \"kubernetes.io/projected/4218688b-c54f-47cc-a11f-8fe7dde5cc48-kube-api-access-m45sr\") pod \"4218688b-c54f-47cc-a11f-8fe7dde5cc48\" (UID: \"4218688b-c54f-47cc-a11f-8fe7dde5cc48\") " Nov 01 06:30:20.295154 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.295037 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4218688b-c54f-47cc-a11f-8fe7dde5cc48-maas-api-service-ca" pod "4218688b-c54f-47cc-a11f-8fe7dde5cc48" (UID: "4218688b-c54f-47cc-a11f-8fe7dde5cc48"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:30:20.295282 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.295244 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4218688b-c54f-47cc-a11f-8fe7dde5cc48-maas-api-service-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:20.297381 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.297346 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4218688b-c54f-47cc-a11f-8fe7dde5cc48-kube-api-access-m45sr" pod "4218688b-c54f-47cc-a11f-8fe7dde5cc48" (UID: "4218688b-c54f-47cc-a11f-8fe7dde5cc48"). InnerVolumeSpecName "kube-api-access-m45sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:30:20.348279 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.348253 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891910-pt7jg" Nov 01 06:30:20.356385 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.356359 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891910-6ngcr"] Nov 01 06:30:20.362407 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.362383 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891910-6ngcr"] Nov 01 06:30:20.395763 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.395727 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1575b2f5-afcc-4d4e-8db0-98e419fced3f-kube-api-access-9stbz\" (UniqueName: \"kubernetes.io/projected/1575b2f5-afcc-4d4e-8db0-98e419fced3f-kube-api-access-9stbz\") pod \"1575b2f5-afcc-4d4e-8db0-98e419fced3f\" (UID: \"1575b2f5-afcc-4d4e-8db0-98e419fced3f\") " Nov 01 06:30:20.395941 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.395868 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1575b2f5-afcc-4d4e-8db0-98e419fced3f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1575b2f5-afcc-4d4e-8db0-98e419fced3f-maas-api-service-ca\") pod \"1575b2f5-afcc-4d4e-8db0-98e419fced3f\" (UID: \"1575b2f5-afcc-4d4e-8db0-98e419fced3f\") " Nov 01 06:30:20.396245 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.396210 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1575b2f5-afcc-4d4e-8db0-98e419fced3f-maas-api-service-ca" pod "1575b2f5-afcc-4d4e-8db0-98e419fced3f" (UID: "1575b2f5-afcc-4d4e-8db0-98e419fced3f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:30:20.396417 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.396392 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m45sr\" (UniqueName: \"kubernetes.io/projected/4218688b-c54f-47cc-a11f-8fe7dde5cc48-kube-api-access-m45sr\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:20.396581 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.396420 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1575b2f5-afcc-4d4e-8db0-98e419fced3f-maas-api-service-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:20.407080 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.407034 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1575b2f5-afcc-4d4e-8db0-98e419fced3f-kube-api-access-9stbz" pod "1575b2f5-afcc-4d4e-8db0-98e419fced3f" (UID: "1575b2f5-afcc-4d4e-8db0-98e419fced3f"). InnerVolumeSpecName "kube-api-access-9stbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:30:20.497563 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.497526 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9stbz\" (UniqueName: \"kubernetes.io/projected/1575b2f5-afcc-4d4e-8db0-98e419fced3f-kube-api-access-9stbz\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:20.857835 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.857745 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-cmplg" Nov 01 06:30:20.857835 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.857774 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-cmplg" event={"ID":"4218688b-c54f-47cc-a11f-8fe7dde5cc48","Type":"ContainerDied","Data":"37dc60be50ff8ee0ac23f679394c72660939327ec54a981bbd5d67979d118a72"} Nov 01 06:30:20.858060 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.857847 5040 scope.go:122] "RemoveContainer" containerID="cae240eef2073b3501cb94754f84e55c1dbc88696ba254f01fccb9948b67e45e" Nov 01 06:30:20.860796 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.860766 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891910-pt7jg" Nov 01 06:30:20.860925 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.860801 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891910-pt7jg" event={"ID":"1575b2f5-afcc-4d4e-8db0-98e419fced3f","Type":"ContainerDied","Data":"a2a55072832f368ebae2104c1995b7b6bea08b6a95fdfbcd06a17baceb03c418"} Nov 01 06:30:20.870233 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.870211 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-745a18-29891910-gw4vg" event={"ID":"aecbbf55-6343-4752-8e2b-0f3771c4df57","Type":"ContainerDied","Data":"a0d1891b9b81c99d803f97e17c4b9a92066297b5ba6d2aa165ce9d70eb3c6235"} Nov 01 06:30:20.870343 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.870241 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-745a18-29891910-gw4vg" Nov 01 06:30:20.871298 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.871281 5040 scope.go:122] "RemoveContainer" containerID="86d0b25dcfbc7775d23d802c985e77c9df389dcdbb1f18b5bc1a682b1348d902" Nov 01 06:30:20.881388 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:20.881328 5040 scope.go:122] "RemoveContainer" containerID="5e88d34db60185a8e3b5b76d768ac5a0214145212d713c2a16066c5a9382c69a" Nov 01 06:30:21.025062 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:21.024995 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891910-pt7jg"] Nov 01 06:30:21.029075 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:21.029046 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891910-pt7jg"] Nov 01 06:30:21.137445 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:21.137410 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891910-4x68f"] Nov 01 06:30:21.142460 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:21.142432 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891910-cmplg"] Nov 01 06:30:21.147556 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:21.147531 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891910-cmplg"] Nov 01 06:30:21.258615 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:21.258232 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-745a18-29891910-gw4vg"] Nov 01 06:30:21.261775 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:21.261752 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-745a18-29891910-gw4vg"] Nov 01 06:30:21.276629 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:21.276604 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891910-4x68f" Nov 01 06:30:21.307153 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:21.307117 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1da58795-e0ae-4db3-9774-0c8d997352a6-kube-api-access-6pjnd\" (UniqueName: \"kubernetes.io/projected/1da58795-e0ae-4db3-9774-0c8d997352a6-kube-api-access-6pjnd\") pod \"1da58795-e0ae-4db3-9774-0c8d997352a6\" (UID: \"1da58795-e0ae-4db3-9774-0c8d997352a6\") " Nov 01 06:30:21.307574 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:21.307276 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1da58795-e0ae-4db3-9774-0c8d997352a6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da58795-e0ae-4db3-9774-0c8d997352a6-maas-api-service-ca\") pod \"1da58795-e0ae-4db3-9774-0c8d997352a6\" (UID: \"1da58795-e0ae-4db3-9774-0c8d997352a6\") " Nov 01 06:30:21.307709 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:21.307687 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1da58795-e0ae-4db3-9774-0c8d997352a6-maas-api-service-ca" pod "1da58795-e0ae-4db3-9774-0c8d997352a6" (UID: "1da58795-e0ae-4db3-9774-0c8d997352a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:30:21.309278 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:21.309257 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1da58795-e0ae-4db3-9774-0c8d997352a6-kube-api-access-6pjnd" pod "1da58795-e0ae-4db3-9774-0c8d997352a6" (UID: "1da58795-e0ae-4db3-9774-0c8d997352a6"). InnerVolumeSpecName "kube-api-access-6pjnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:30:21.409013 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:21.408936 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da58795-e0ae-4db3-9774-0c8d997352a6-maas-api-service-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:21.409013 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:21.408967 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pjnd\" (UniqueName: \"kubernetes.io/projected/1da58795-e0ae-4db3-9774-0c8d997352a6-kube-api-access-6pjnd\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:21.487545 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:21.487508 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891910-cn5cf"] Nov 01 06:30:21.516976 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:21.516943 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1575b2f5-afcc-4d4e-8db0-98e419fced3f" path="/var/lib/kubelet/pods/1575b2f5-afcc-4d4e-8db0-98e419fced3f/volumes" Nov 01 06:30:21.517359 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:21.517346 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15896689-4508-43e1-95b2-df12c0853cd6" path="/var/lib/kubelet/pods/15896689-4508-43e1-95b2-df12c0853cd6/volumes" Nov 01 06:30:21.517738 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:21.517725 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4218688b-c54f-47cc-a11f-8fe7dde5cc48" path="/var/lib/kubelet/pods/4218688b-c54f-47cc-a11f-8fe7dde5cc48/volumes" Nov 01 06:30:21.518396 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:21.518382 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98183e7b-70c1-4af0-9b64-9b4cba9424a1" path="/var/lib/kubelet/pods/98183e7b-70c1-4af0-9b64-9b4cba9424a1/volumes" Nov 01 06:30:21.518782 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:21.518769 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aecbbf55-6343-4752-8e2b-0f3771c4df57" path="/var/lib/kubelet/pods/aecbbf55-6343-4752-8e2b-0f3771c4df57/volumes" Nov 01 06:30:21.519124 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:21.519113 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd27d740-cb57-479b-97ca-517a09d689b3" path="/var/lib/kubelet/pods/dd27d740-cb57-479b-97ca-517a09d689b3/volumes" Nov 01 06:30:21.644800 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:21.644775 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891910-cn5cf" Nov 01 06:30:21.712398 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:21.712308 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66ba8fe2-b2da-4281-aec7-7f2b8440e548-kube-api-access-v6gg5\" (UniqueName: \"kubernetes.io/projected/66ba8fe2-b2da-4281-aec7-7f2b8440e548-kube-api-access-v6gg5\") pod \"66ba8fe2-b2da-4281-aec7-7f2b8440e548\" (UID: \"66ba8fe2-b2da-4281-aec7-7f2b8440e548\") " Nov 01 06:30:21.712398 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:21.712346 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66ba8fe2-b2da-4281-aec7-7f2b8440e548-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66ba8fe2-b2da-4281-aec7-7f2b8440e548-maas-api-service-ca\") pod \"66ba8fe2-b2da-4281-aec7-7f2b8440e548\" (UID: \"66ba8fe2-b2da-4281-aec7-7f2b8440e548\") " Nov 01 06:30:21.712846 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:21.712819 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ba8fe2-b2da-4281-aec7-7f2b8440e548-maas-api-service-ca" pod "66ba8fe2-b2da-4281-aec7-7f2b8440e548" (UID: "66ba8fe2-b2da-4281-aec7-7f2b8440e548"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:30:21.714589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:21.714567 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ba8fe2-b2da-4281-aec7-7f2b8440e548-kube-api-access-v6gg5" pod "66ba8fe2-b2da-4281-aec7-7f2b8440e548" (UID: "66ba8fe2-b2da-4281-aec7-7f2b8440e548"). InnerVolumeSpecName "kube-api-access-v6gg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:30:21.814365 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:21.814326 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6gg5\" (UniqueName: \"kubernetes.io/projected/66ba8fe2-b2da-4281-aec7-7f2b8440e548-kube-api-access-v6gg5\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:21.814365 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:21.814366 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66ba8fe2-b2da-4281-aec7-7f2b8440e548-maas-api-service-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:21.880377 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:21.880352 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891910-cn5cf" Nov 01 06:30:21.880559 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:21.880392 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891910-cn5cf" event={"ID":"66ba8fe2-b2da-4281-aec7-7f2b8440e548","Type":"ContainerDied","Data":"cb8d9717cbc13a9667e37537f53d997938b37c2b63088b2f5bec7b5ffad82692"} Nov 01 06:30:21.880559 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:21.880461 5040 scope.go:122] "RemoveContainer" containerID="67f43db529f5d6d7c093d494fa61a05eeeff29d207e0c9556fb2d481a7d85e9d" Nov 01 06:30:21.882022 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:21.881997 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891910-4x68f" Nov 01 06:30:21.882022 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:21.882010 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891910-4x68f" event={"ID":"1da58795-e0ae-4db3-9774-0c8d997352a6","Type":"ContainerDied","Data":"7e0d2866cb201ce7b14c378c0e9eff71639152daaea426eeb525b0e75486840a"} Nov 01 06:30:21.891402 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:21.891386 5040 scope.go:122] "RemoveContainer" containerID="45542b10ecac91670fe3f2b7f347376c0dab918bff29207b7d77073ae3386934" Nov 01 06:30:22.016023 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:22.015993 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 06:30:22.017706 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:22.017681 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 06:30:22.047274 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:22.047232 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891910-cn5cf"] Nov 01 06:30:22.050704 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:22.050678 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891910-cn5cf"] Nov 01 06:30:22.163964 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:22.163925 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891910-4x68f"] Nov 01 06:30:22.168355 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:22.168327 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891910-4x68f"] Nov 01 06:30:22.979890 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:22.979862 5040 scope.go:122] "RemoveContainer" containerID="2f05ad9a69e7c18c9012fde3fe410d58abdd5b786ed42fd1c92e3a194e2a7957" Nov 01 06:30:22.989734 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:22.989709 5040 scope.go:122] "RemoveContainer" containerID="07e441a980a384d081582a0589e20364ca5014d55becb97854664f3c3f3cf182" Nov 01 06:30:22.998987 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:22.998965 5040 scope.go:122] "RemoveContainer" containerID="86a27d0a0902b325e907436854e28ea824af1bc1eafc159336eb8e476b5c257f" Nov 01 06:30:23.008540 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:23.008515 5040 scope.go:122] "RemoveContainer" containerID="fa366863a122d0c5cc4a9303cb729cb4f2ab0dcfdda9cdfb0942429339e3e8a9" Nov 01 06:30:23.515401 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:23.515369 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1da58795-e0ae-4db3-9774-0c8d997352a6" path="/var/lib/kubelet/pods/1da58795-e0ae-4db3-9774-0c8d997352a6/volumes" Nov 01 06:30:23.515822 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:23.515808 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66ba8fe2-b2da-4281-aec7-7f2b8440e548" path="/var/lib/kubelet/pods/66ba8fe2-b2da-4281-aec7-7f2b8440e548/volumes" Nov 01 06:30:24.048151 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:24.048117 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfca6de2-vdpw6"] Nov 01 06:30:24.098498 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:24.098453 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfca6de2-vdpw6"] Nov 01 06:30:25.515027 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:25.514995 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c45608df-359e-41fb-90c7-917b59e9042c" path="/var/lib/kubelet/pods/c45608df-359e-41fb-90c7-917b59e9042c/volumes" Nov 01 06:30:26.047674 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:26.047638 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddab543d-xn7hm"] Nov 01 06:30:26.098131 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:26.098090 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddab543d-xn7hm"] Nov 01 06:30:27.514844 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:27.514810 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eec1ba8-ec11-49df-b49f-8beab1f82be5" path="/var/lib/kubelet/pods/0eec1ba8-ec11-49df-b49f-8beab1f82be5/volumes" Nov 01 06:30:39.973987 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:30:39.973947 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:30:45.050390 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:45.050353 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-618a5fc8-jb24m"] Nov 01 06:30:45.099971 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:45.099937 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-618a5fc8-jb24m"] Nov 01 06:30:45.515974 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:45.515944 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbdd1108-90e1-4806-a56a-b1e310fdc62d" path="/var/lib/kubelet/pods/dbdd1108-90e1-4806-a56a-b1e310fdc62d/volumes" Nov 01 06:30:52.047771 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:52.047730 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09c6fd53-gtn2n"] Nov 01 06:30:52.097961 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:52.097926 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09c6fd53-gtn2n"] Nov 01 06:30:53.514998 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:53.514960 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe3c4dcf-3d99-4132-ba43-d177632a6a2c" path="/var/lib/kubelet/pods/fe3c4dcf-3d99-4132-ba43-d177632a6a2c/volumes" Nov 01 06:30:57.168898 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.168860 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq"] Nov 01 06:30:57.169407 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169350 5040 cpu_manager.go:414] "containerMap: removing container" podUID="a282a96e-d0fc-4d97-b2e1-35b8c593bcfc" containerName="cleanup" Nov 01 06:30:57.169407 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169364 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a282a96e-d0fc-4d97-b2e1-35b8c593bcfc" containerName="cleanup" Nov 01 06:30:57.169407 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169380 5040 cpu_manager.go:414] "containerMap: removing container" podUID="98183e7b-70c1-4af0-9b64-9b4cba9424a1" containerName="cleanup" Nov 01 06:30:57.169407 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169385 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98183e7b-70c1-4af0-9b64-9b4cba9424a1" containerName="cleanup" Nov 01 06:30:57.169407 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169392 5040 cpu_manager.go:414] "containerMap: removing container" podUID="1575b2f5-afcc-4d4e-8db0-98e419fced3f" containerName="cleanup" Nov 01 06:30:57.169407 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169397 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1575b2f5-afcc-4d4e-8db0-98e419fced3f" containerName="cleanup" Nov 01 06:30:57.169407 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169404 5040 cpu_manager.go:414] "containerMap: removing container" podUID="15896689-4508-43e1-95b2-df12c0853cd6" containerName="cleanup" Nov 01 06:30:57.169407 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169409 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15896689-4508-43e1-95b2-df12c0853cd6" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169415 5040 cpu_manager.go:414] "containerMap: removing container" podUID="dd27d740-cb57-479b-97ca-517a09d689b3" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169420 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd27d740-cb57-479b-97ca-517a09d689b3" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169430 5040 cpu_manager.go:414] "containerMap: removing container" podUID="1575b2f5-afcc-4d4e-8db0-98e419fced3f" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169434 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1575b2f5-afcc-4d4e-8db0-98e419fced3f" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169441 5040 cpu_manager.go:414] "containerMap: removing container" podUID="1da58795-e0ae-4db3-9774-0c8d997352a6" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169446 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1da58795-e0ae-4db3-9774-0c8d997352a6" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169452 5040 cpu_manager.go:414] "containerMap: removing container" podUID="1da58795-e0ae-4db3-9774-0c8d997352a6" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169456 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1da58795-e0ae-4db3-9774-0c8d997352a6" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169463 5040 cpu_manager.go:414] "containerMap: removing container" podUID="1575b2f5-afcc-4d4e-8db0-98e419fced3f" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169468 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1575b2f5-afcc-4d4e-8db0-98e419fced3f" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169491 5040 cpu_manager.go:414] "containerMap: removing container" podUID="aecbbf55-6343-4752-8e2b-0f3771c4df57" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169499 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aecbbf55-6343-4752-8e2b-0f3771c4df57" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169511 5040 cpu_manager.go:414] "containerMap: removing container" podUID="66ba8fe2-b2da-4281-aec7-7f2b8440e548" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169515 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66ba8fe2-b2da-4281-aec7-7f2b8440e548" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169523 5040 cpu_manager.go:414] "containerMap: removing container" podUID="4218688b-c54f-47cc-a11f-8fe7dde5cc48" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169527 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4218688b-c54f-47cc-a11f-8fe7dde5cc48" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169535 5040 cpu_manager.go:414] "containerMap: removing container" podUID="7c7c88e9-2475-4bf5-9c1a-426c1643312c" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169539 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c7c88e9-2475-4bf5-9c1a-426c1643312c" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169549 5040 cpu_manager.go:414] "containerMap: removing container" podUID="a282a96e-d0fc-4d97-b2e1-35b8c593bcfc" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169557 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a282a96e-d0fc-4d97-b2e1-35b8c593bcfc" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169566 5040 cpu_manager.go:414] "containerMap: removing container" podUID="66ba8fe2-b2da-4281-aec7-7f2b8440e548" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169570 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66ba8fe2-b2da-4281-aec7-7f2b8440e548" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169575 5040 cpu_manager.go:414] "containerMap: removing container" podUID="98183e7b-70c1-4af0-9b64-9b4cba9424a1" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169579 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98183e7b-70c1-4af0-9b64-9b4cba9424a1" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169586 5040 cpu_manager.go:414] "containerMap: removing container" podUID="1da58795-e0ae-4db3-9774-0c8d997352a6" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169590 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1da58795-e0ae-4db3-9774-0c8d997352a6" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169598 5040 cpu_manager.go:414] "containerMap: removing container" podUID="98183e7b-70c1-4af0-9b64-9b4cba9424a1" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169602 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98183e7b-70c1-4af0-9b64-9b4cba9424a1" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169609 5040 cpu_manager.go:414] "containerMap: removing container" podUID="15896689-4508-43e1-95b2-df12c0853cd6" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169613 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15896689-4508-43e1-95b2-df12c0853cd6" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169619 5040 cpu_manager.go:414] "containerMap: removing container" podUID="aecbbf55-6343-4752-8e2b-0f3771c4df57" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169622 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aecbbf55-6343-4752-8e2b-0f3771c4df57" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169632 5040 cpu_manager.go:414] "containerMap: removing container" podUID="4218688b-c54f-47cc-a11f-8fe7dde5cc48" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169639 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4218688b-c54f-47cc-a11f-8fe7dde5cc48" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169650 5040 cpu_manager.go:414] "containerMap: removing container" podUID="7c7c88e9-2475-4bf5-9c1a-426c1643312c" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169656 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c7c88e9-2475-4bf5-9c1a-426c1643312c" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169665 5040 cpu_manager.go:414] "containerMap: removing container" podUID="66ba8fe2-b2da-4281-aec7-7f2b8440e548" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169671 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66ba8fe2-b2da-4281-aec7-7f2b8440e548" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169688 5040 cpu_manager.go:414] "containerMap: removing container" podUID="dd27d740-cb57-479b-97ca-517a09d689b3" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169693 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd27d740-cb57-479b-97ca-517a09d689b3" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169779 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c7c88e9-2475-4bf5-9c1a-426c1643312c" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169788 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd27d740-cb57-479b-97ca-517a09d689b3" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169795 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="4218688b-c54f-47cc-a11f-8fe7dde5cc48" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169800 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="aecbbf55-6343-4752-8e2b-0f3771c4df57" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169805 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="a282a96e-d0fc-4d97-b2e1-35b8c593bcfc" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169812 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="98183e7b-70c1-4af0-9b64-9b4cba9424a1" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169818 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="1575b2f5-afcc-4d4e-8db0-98e419fced3f" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169823 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="15896689-4508-43e1-95b2-df12c0853cd6" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169828 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd27d740-cb57-479b-97ca-517a09d689b3" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169833 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="1575b2f5-afcc-4d4e-8db0-98e419fced3f" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169838 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="1da58795-e0ae-4db3-9774-0c8d997352a6" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169843 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="1da58795-e0ae-4db3-9774-0c8d997352a6" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169849 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="1575b2f5-afcc-4d4e-8db0-98e419fced3f" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169856 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="aecbbf55-6343-4752-8e2b-0f3771c4df57" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169861 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="15896689-4508-43e1-95b2-df12c0853cd6" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169867 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="66ba8fe2-b2da-4281-aec7-7f2b8440e548" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169873 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="a282a96e-d0fc-4d97-b2e1-35b8c593bcfc" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169879 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="a282a96e-d0fc-4d97-b2e1-35b8c593bcfc" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169884 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="66ba8fe2-b2da-4281-aec7-7f2b8440e548" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169889 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="1da58795-e0ae-4db3-9774-0c8d997352a6" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169896 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="98183e7b-70c1-4af0-9b64-9b4cba9424a1" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169901 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="15896689-4508-43e1-95b2-df12c0853cd6" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169907 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="4218688b-c54f-47cc-a11f-8fe7dde5cc48" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169913 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c7c88e9-2475-4bf5-9c1a-426c1643312c" containerName="cleanup" Nov 01 06:30:57.169907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.169917 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="66ba8fe2-b2da-4281-aec7-7f2b8440e548" containerName="cleanup" Nov 01 06:30:57.173585 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.173563 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:30:57.297501 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.296622 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-747b7b-openshift-default-dockercfg-rzs7z\"" Nov 01 06:30:57.301221 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.301192 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq"] Nov 01 06:30:57.394579 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.394539 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14b4da92-5164-45df-8306-5f228a7c5ac2-istio-envoy\") pod \"e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq\" (UID: \"14b4da92-5164-45df-8306-5f228a7c5ac2\") " pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:30:57.394885 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.394866 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14b4da92-5164-45df-8306-5f228a7c5ac2-istio-data\") pod \"e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq\" (UID: \"14b4da92-5164-45df-8306-5f228a7c5ac2\") " pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:30:57.395032 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.395013 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14b4da92-5164-45df-8306-5f228a7c5ac2-istiod-ca-cert\") pod \"e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq\" (UID: \"14b4da92-5164-45df-8306-5f228a7c5ac2\") " pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:30:57.395207 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.395192 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14b4da92-5164-45df-8306-5f228a7c5ac2-workload-certs\") pod \"e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq\" (UID: \"14b4da92-5164-45df-8306-5f228a7c5ac2\") " pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:30:57.395345 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.395330 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b4da92-5164-45df-8306-5f228a7c5ac2-workload-socket\") pod \"e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq\" (UID: \"14b4da92-5164-45df-8306-5f228a7c5ac2\") " pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:30:57.395509 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.395491 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgprk\" (UniqueName: \"kubernetes.io/projected/14b4da92-5164-45df-8306-5f228a7c5ac2-kube-api-access-vgprk\") pod \"e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq\" (UID: \"14b4da92-5164-45df-8306-5f228a7c5ac2\") " pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:30:57.395770 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.395752 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b4da92-5164-45df-8306-5f228a7c5ac2-credential-socket\") pod \"e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq\" (UID: \"14b4da92-5164-45df-8306-5f228a7c5ac2\") " pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:30:57.395969 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.395952 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14b4da92-5164-45df-8306-5f228a7c5ac2-istio-podinfo\") pod \"e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq\" (UID: \"14b4da92-5164-45df-8306-5f228a7c5ac2\") " pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:30:57.396124 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.396109 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14b4da92-5164-45df-8306-5f228a7c5ac2-istio-token\") pod \"e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq\" (UID: \"14b4da92-5164-45df-8306-5f228a7c5ac2\") " pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:30:57.497540 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.497473 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14b4da92-5164-45df-8306-5f228a7c5ac2-istio-data\") pod \"e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq\" (UID: \"14b4da92-5164-45df-8306-5f228a7c5ac2\") " pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:30:57.497738 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.497620 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14b4da92-5164-45df-8306-5f228a7c5ac2-istiod-ca-cert\") pod \"e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq\" (UID: \"14b4da92-5164-45df-8306-5f228a7c5ac2\") " pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:30:57.497738 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.497662 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14b4da92-5164-45df-8306-5f228a7c5ac2-workload-certs\") pod \"e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq\" (UID: \"14b4da92-5164-45df-8306-5f228a7c5ac2\") " pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:30:57.497738 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.497700 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b4da92-5164-45df-8306-5f228a7c5ac2-workload-socket\") pod \"e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq\" (UID: \"14b4da92-5164-45df-8306-5f228a7c5ac2\") " pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:30:57.497921 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.497749 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgprk\" (UniqueName: \"kubernetes.io/projected/14b4da92-5164-45df-8306-5f228a7c5ac2-kube-api-access-vgprk\") pod \"e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq\" (UID: \"14b4da92-5164-45df-8306-5f228a7c5ac2\") " pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:30:57.497921 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.497797 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b4da92-5164-45df-8306-5f228a7c5ac2-credential-socket\") pod \"e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq\" (UID: \"14b4da92-5164-45df-8306-5f228a7c5ac2\") " pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:30:57.497921 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.497876 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14b4da92-5164-45df-8306-5f228a7c5ac2-istio-podinfo\") pod \"e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq\" (UID: \"14b4da92-5164-45df-8306-5f228a7c5ac2\") " pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:30:57.497921 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.497899 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14b4da92-5164-45df-8306-5f228a7c5ac2-istio-token\") pod \"e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq\" (UID: \"14b4da92-5164-45df-8306-5f228a7c5ac2\") " pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:30:57.498151 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.497919 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14b4da92-5164-45df-8306-5f228a7c5ac2-istio-data\") pod \"e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq\" (UID: \"14b4da92-5164-45df-8306-5f228a7c5ac2\") " pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:30:57.498151 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.497947 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14b4da92-5164-45df-8306-5f228a7c5ac2-istio-envoy\") pod \"e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq\" (UID: \"14b4da92-5164-45df-8306-5f228a7c5ac2\") " pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:30:57.498151 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.498030 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14b4da92-5164-45df-8306-5f228a7c5ac2-workload-certs\") pod \"e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq\" (UID: \"14b4da92-5164-45df-8306-5f228a7c5ac2\") " pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:30:57.498151 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.498119 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b4da92-5164-45df-8306-5f228a7c5ac2-workload-socket\") pod \"e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq\" (UID: \"14b4da92-5164-45df-8306-5f228a7c5ac2\") " pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:30:57.498444 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.498337 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b4da92-5164-45df-8306-5f228a7c5ac2-credential-socket\") pod \"e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq\" (UID: \"14b4da92-5164-45df-8306-5f228a7c5ac2\") " pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:30:57.498539 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.498472 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14b4da92-5164-45df-8306-5f228a7c5ac2-istiod-ca-cert\") pod \"e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq\" (UID: \"14b4da92-5164-45df-8306-5f228a7c5ac2\") " pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:30:57.500203 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.500181 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14b4da92-5164-45df-8306-5f228a7c5ac2-istio-envoy\") pod \"e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq\" (UID: \"14b4da92-5164-45df-8306-5f228a7c5ac2\") " pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:30:57.500859 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.500834 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14b4da92-5164-45df-8306-5f228a7c5ac2-istio-podinfo\") pod \"e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq\" (UID: \"14b4da92-5164-45df-8306-5f228a7c5ac2\") " pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:30:57.559651 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.559611 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14b4da92-5164-45df-8306-5f228a7c5ac2-istio-token\") pod \"e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq\" (UID: \"14b4da92-5164-45df-8306-5f228a7c5ac2\") " pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:30:57.560331 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.560303 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgprk\" (UniqueName: \"kubernetes.io/projected/14b4da92-5164-45df-8306-5f228a7c5ac2-kube-api-access-vgprk\") pod \"e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq\" (UID: \"14b4da92-5164-45df-8306-5f228a7c5ac2\") " pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:30:57.787304 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:57.787212 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:30:58.223648 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:58.223612 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq"] Nov 01 06:30:58.224116 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:30:58.224090 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b4da92_5164_45df_8306_5f228a7c5ac2.slice/crio-bc39b2531dfc52449ea4c5ea2c0dc6feb32d7ada5e42f678ed37167dc5ef5589 WatchSource:0}: Error finding container bc39b2531dfc52449ea4c5ea2c0dc6feb32d7ada5e42f678ed37167dc5ef5589: Status 404 returned error can't find the container with id bc39b2531dfc52449ea4c5ea2c0dc6feb32d7ada5e42f678ed37167dc5ef5589 Nov 01 06:30:58.226635 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:58.226599 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:30:58.226734 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:58.226667 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:30:58.226734 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:58.226697 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:30:59.128323 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:59.128274 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" event={"ID":"14b4da92-5164-45df-8306-5f228a7c5ac2","Type":"ContainerStarted","Data":"ff3ac7610ee3e1a638c58bed2ebc6b8eff09ecc403f1b740651aba81e65c6228"} Nov 01 06:30:59.128527 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:59.128378 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" event={"ID":"14b4da92-5164-45df-8306-5f228a7c5ac2","Type":"ContainerStarted","Data":"bc39b2531dfc52449ea4c5ea2c0dc6feb32d7ada5e42f678ed37167dc5ef5589"} Nov 01 06:30:59.301441 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:59.301384 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" podStartSLOduration=2.301369566 podStartE2EDuration="2.301369566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:59.29952783 +0000 UTC m=+2737.954644994" watchObservedRunningTime="2026-11-01 06:30:59.301369566 +0000 UTC m=+2737.956486784" Nov 01 06:30:59.787380 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:59.787340 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:30:59.793012 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:30:59.792986 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:31:00.135515 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:00.135418 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:31:00.136569 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:00.136537 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:31:03.048508 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:03.048443 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a19f8bb-slrvd"] Nov 01 06:31:03.099891 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:03.099860 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a19f8bb-slrvd"] Nov 01 06:31:03.515030 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:03.514995 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92f94f44-2b25-4809-a38d-e3eabab49544" path="/var/lib/kubelet/pods/92f94f44-2b25-4809-a38d-e3eabab49544/volumes" Nov 01 06:31:09.973775 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:31:09.973727 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:31:17.048003 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:17.047964 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac6348fe-zvftk"] Nov 01 06:31:17.097778 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:17.097746 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac6348fe-zvftk"] Nov 01 06:31:17.526896 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:17.526855 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25fdfaf7-a907-4d2e-9786-93a5db503d2b" path="/var/lib/kubelet/pods/25fdfaf7-a907-4d2e-9786-93a5db503d2b/volumes" Nov 01 06:31:23.102749 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:23.102717 5040 scope.go:122] "RemoveContainer" containerID="13268b14f96b56df34126da9bf1dfbf076aa77033cbdb4ccb35184edf7fbe6a8" Nov 01 06:31:23.113937 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:23.113299 5040 scope.go:122] "RemoveContainer" containerID="d9ce6f2cd7dc74f36882d98c9d340cfe3b3f0625b057fcbcb7e3f3dcd7d91ca7" Nov 01 06:31:23.126159 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:23.126132 5040 scope.go:122] "RemoveContainer" containerID="4c5e3a82f22ee8291cbaf4ca5d2b212439236a9c910784036c2a42c812250358" Nov 01 06:31:23.135467 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:23.135442 5040 scope.go:122] "RemoveContainer" containerID="439f64f5f5c1c793062a51001281da16a2cf5a52d4bd9a5b76af9bb80ec4725d" Nov 01 06:31:26.040974 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.040933 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv"] Nov 01 06:31:26.041452 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.041410 5040 cpu_manager.go:414] "containerMap: removing container" podUID="a282a96e-d0fc-4d97-b2e1-35b8c593bcfc" containerName="cleanup" Nov 01 06:31:26.041452 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.041422 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a282a96e-d0fc-4d97-b2e1-35b8c593bcfc" containerName="cleanup" Nov 01 06:31:26.041452 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.041439 5040 cpu_manager.go:414] "containerMap: removing container" podUID="15896689-4508-43e1-95b2-df12c0853cd6" containerName="cleanup" Nov 01 06:31:26.041452 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.041444 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15896689-4508-43e1-95b2-df12c0853cd6" containerName="cleanup" Nov 01 06:31:26.041670 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.041458 5040 cpu_manager.go:414] "containerMap: removing container" podUID="aecbbf55-6343-4752-8e2b-0f3771c4df57" containerName="cleanup" Nov 01 06:31:26.041670 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.041463 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aecbbf55-6343-4752-8e2b-0f3771c4df57" containerName="cleanup" Nov 01 06:31:26.041670 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.041474 5040 cpu_manager.go:414] "containerMap: removing container" podUID="4218688b-c54f-47cc-a11f-8fe7dde5cc48" containerName="cleanup" Nov 01 06:31:26.041670 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.041508 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4218688b-c54f-47cc-a11f-8fe7dde5cc48" containerName="cleanup" Nov 01 06:31:26.041670 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.041525 5040 cpu_manager.go:414] "containerMap: removing container" podUID="7c7c88e9-2475-4bf5-9c1a-426c1643312c" containerName="cleanup" Nov 01 06:31:26.041670 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.041529 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c7c88e9-2475-4bf5-9c1a-426c1643312c" containerName="cleanup" Nov 01 06:31:26.041670 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.041542 5040 cpu_manager.go:414] "containerMap: removing container" podUID="dd27d740-cb57-479b-97ca-517a09d689b3" containerName="cleanup" Nov 01 06:31:26.041670 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.041547 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd27d740-cb57-479b-97ca-517a09d689b3" containerName="cleanup" Nov 01 06:31:26.041670 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.041633 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="98183e7b-70c1-4af0-9b64-9b4cba9424a1" containerName="cleanup" Nov 01 06:31:26.041670 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.041641 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c7c88e9-2475-4bf5-9c1a-426c1643312c" containerName="cleanup" Nov 01 06:31:26.041670 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.041648 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd27d740-cb57-479b-97ca-517a09d689b3" containerName="cleanup" Nov 01 06:31:26.041670 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.041653 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="4218688b-c54f-47cc-a11f-8fe7dde5cc48" containerName="cleanup" Nov 01 06:31:26.041670 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.041660 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="aecbbf55-6343-4752-8e2b-0f3771c4df57" containerName="cleanup" Nov 01 06:31:26.046446 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.046428 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv" Nov 01 06:31:26.103224 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.103189 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv"] Nov 01 06:31:26.142430 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.142397 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-home\") pod \"e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv\" (UID: \"72ab782a-6494-430b-a5eb-17f4b4f9ddd3\") " pod="ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv" Nov 01 06:31:26.142650 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.142456 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-model-cache\") pod \"e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv\" (UID: \"72ab782a-6494-430b-a5eb-17f4b4f9ddd3\") " pod="ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv" Nov 01 06:31:26.142650 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.142563 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-tmp-dir\") pod \"e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv\" (UID: \"72ab782a-6494-430b-a5eb-17f4b4f9ddd3\") " pod="ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv" Nov 01 06:31:26.142650 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.142613 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-dshm\") pod \"e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv\" (UID: \"72ab782a-6494-430b-a5eb-17f4b4f9ddd3\") " pod="ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv" Nov 01 06:31:26.142782 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.142699 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws8j9\" (UniqueName: \"kubernetes.io/projected/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-kube-api-access-ws8j9\") pod \"e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv\" (UID: \"72ab782a-6494-430b-a5eb-17f4b4f9ddd3\") " pod="ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv" Nov 01 06:31:26.142782 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.142736 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-tls-certs\") pod \"e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv\" (UID: \"72ab782a-6494-430b-a5eb-17f4b4f9ddd3\") " pod="ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv" Nov 01 06:31:26.150060 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.150030 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c0662f25\"/\"kube-root-ca.crt\"" Nov 01 06:31:26.150195 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.150170 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c0662f25\"/\"openshift-service-ca.crt\"" Nov 01 06:31:26.150195 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.150182 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c0662f25\"/\"e2e-lifecycle-model-c0662f25-kserve-self-signed-certs\"" Nov 01 06:31:26.150195 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.150170 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c0662f25\"/\"default-dockercfg-48vvq\"" Nov 01 06:31:26.244010 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.243978 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-home\") pod \"e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv\" (UID: \"72ab782a-6494-430b-a5eb-17f4b4f9ddd3\") " pod="ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv" Nov 01 06:31:26.244209 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.244046 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-model-cache\") pod \"e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv\" (UID: \"72ab782a-6494-430b-a5eb-17f4b4f9ddd3\") " pod="ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv" Nov 01 06:31:26.244209 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.244071 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-tmp-dir\") pod \"e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv\" (UID: \"72ab782a-6494-430b-a5eb-17f4b4f9ddd3\") " pod="ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv" Nov 01 06:31:26.244209 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.244106 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-dshm\") pod \"e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv\" (UID: \"72ab782a-6494-430b-a5eb-17f4b4f9ddd3\") " pod="ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv" Nov 01 06:31:26.244209 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.244164 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws8j9\" (UniqueName: \"kubernetes.io/projected/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-kube-api-access-ws8j9\") pod \"e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv\" (UID: \"72ab782a-6494-430b-a5eb-17f4b4f9ddd3\") " pod="ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv" Nov 01 06:31:26.244209 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.244202 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-tls-certs\") pod \"e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv\" (UID: \"72ab782a-6494-430b-a5eb-17f4b4f9ddd3\") " pod="ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv" Nov 01 06:31:26.244511 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.244461 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-home\") pod \"e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv\" (UID: \"72ab782a-6494-430b-a5eb-17f4b4f9ddd3\") " pod="ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv" Nov 01 06:31:26.244558 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.244515 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-tmp-dir\") pod \"e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv\" (UID: \"72ab782a-6494-430b-a5eb-17f4b4f9ddd3\") " pod="ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv" Nov 01 06:31:26.244614 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.244578 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-model-cache\") pod \"e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv\" (UID: \"72ab782a-6494-430b-a5eb-17f4b4f9ddd3\") " pod="ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv" Nov 01 06:31:26.246634 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.246605 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-dshm\") pod \"e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv\" (UID: \"72ab782a-6494-430b-a5eb-17f4b4f9ddd3\") " pod="ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv" Nov 01 06:31:26.246818 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.246800 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-tls-certs\") pod \"e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv\" (UID: \"72ab782a-6494-430b-a5eb-17f4b4f9ddd3\") " pod="ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv" Nov 01 06:31:26.308907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.308821 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws8j9\" (UniqueName: \"kubernetes.io/projected/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-kube-api-access-ws8j9\") pod \"e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv\" (UID: \"72ab782a-6494-430b-a5eb-17f4b4f9ddd3\") " pod="ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv" Nov 01 06:31:26.358827 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.358789 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv" Nov 01 06:31:26.742678 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:26.742645 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv"] Nov 01 06:31:26.743702 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:31:26.743675 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72ab782a_6494_430b_a5eb_17f4b4f9ddd3.slice/crio-a6a25b0f3c63c12bd11dd135ab33dcabc0496dfa05ec3f97f813d6be57f36ea5 WatchSource:0}: Error finding container a6a25b0f3c63c12bd11dd135ab33dcabc0496dfa05ec3f97f813d6be57f36ea5: Status 404 returned error can't find the container with id a6a25b0f3c63c12bd11dd135ab33dcabc0496dfa05ec3f97f813d6be57f36ea5 Nov 01 06:31:27.328057 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:27.328016 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv" event={"ID":"72ab782a-6494-430b-a5eb-17f4b4f9ddd3","Type":"ContainerStarted","Data":"390d5f45fdf9571d0d91af0185ecbd9f98c5914de8a5af1f15c4cb136f732c3f"} Nov 01 06:31:27.328540 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:27.328070 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv" event={"ID":"72ab782a-6494-430b-a5eb-17f4b4f9ddd3","Type":"ContainerStarted","Data":"a6a25b0f3c63c12bd11dd135ab33dcabc0496dfa05ec3f97f813d6be57f36ea5"} Nov 01 06:31:27.359272 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:27.359224 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv" Nov 01 06:31:27.446874 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:27.446817 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv" podStartSLOduration=2.446803053 podStartE2EDuration="2.446803053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:31:27.445828709 +0000 UTC m=+2766.100945878" watchObservedRunningTime="2026-11-01 06:31:27.446803053 +0000 UTC m=+2766.101920216" Nov 01 06:31:30.049346 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:30.049306 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89596147-8d6s6"] Nov 01 06:31:30.053211 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:30.053189 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89596147-8d6s6"] Nov 01 06:31:31.514621 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:31.514591 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cf31556-92b7-491c-b685-987134f8282d" path="/var/lib/kubelet/pods/8cf31556-92b7-491c-b685-987134f8282d/volumes" Nov 01 06:31:36.359047 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:36.359005 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv" Nov 01 06:31:36.372607 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:36.372579 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv" Nov 01 06:31:36.404752 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:36.404721 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv" Nov 01 06:31:39.973007 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:31:39.972963 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:31:42.600857 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:42.600825 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt"] Nov 01 06:31:42.601331 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:42.601026 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" podUID="068716fa-4615-4dbf-9c5f-41687865d5b1" containerName="istio-proxy" containerID="cri-o://3b141c3d9dc299bf597a555cd2cfdc2c9af0a7d6b9a3b4d1970e729214da666d" gracePeriod=30 Nov 01 06:31:44.473990 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:44.473957 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aed62e2-kksd4"] Nov 01 06:31:44.479230 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:44.479204 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aed62e2-kksd4" Nov 01 06:31:44.536174 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:44.536136 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aed62e2-kksd4"] Nov 01 06:31:44.558688 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:44.558654 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:31:44.558688 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:44.558683 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:31:44.558903 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:44.558694 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:31:44.563179 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:44.563148 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c08b85-19ce-404f-b630-8dda894d9d81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-0aed62e2-kksd4\" (UID: \"b8c08b85-19ce-404f-b630-8dda894d9d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aed62e2-kksd4" Nov 01 06:31:44.563348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:44.563321 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq7vl\" (UniqueName: \"kubernetes.io/projected/b8c08b85-19ce-404f-b630-8dda894d9d81-kube-api-access-tq7vl\") pod \"maas-api-revoke-keys-e2e-derive-0aed62e2-kksd4\" (UID: \"b8c08b85-19ce-404f-b630-8dda894d9d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aed62e2-kksd4" Nov 01 06:31:44.563419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:44.563372 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c08b85-19ce-404f-b630-8dda894d9d81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-0aed62e2-kksd4\" (UID: \"b8c08b85-19ce-404f-b630-8dda894d9d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aed62e2-kksd4" Nov 01 06:31:44.582375 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:44.582351 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:31:44.664709 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:44.664679 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq7vl\" (UniqueName: \"kubernetes.io/projected/b8c08b85-19ce-404f-b630-8dda894d9d81-kube-api-access-tq7vl\") pod \"maas-api-revoke-keys-e2e-derive-0aed62e2-kksd4\" (UID: \"b8c08b85-19ce-404f-b630-8dda894d9d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aed62e2-kksd4" Nov 01 06:31:44.664917 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:44.664716 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c08b85-19ce-404f-b630-8dda894d9d81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-0aed62e2-kksd4\" (UID: \"b8c08b85-19ce-404f-b630-8dda894d9d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aed62e2-kksd4" Nov 01 06:31:44.664917 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:44.664831 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c08b85-19ce-404f-b630-8dda894d9d81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-0aed62e2-kksd4\" (UID: \"b8c08b85-19ce-404f-b630-8dda894d9d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aed62e2-kksd4" Nov 01 06:31:44.665579 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:44.665553 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c08b85-19ce-404f-b630-8dda894d9d81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-0aed62e2-kksd4\" (UID: \"b8c08b85-19ce-404f-b630-8dda894d9d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aed62e2-kksd4" Nov 01 06:31:44.667317 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:44.667292 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c08b85-19ce-404f-b630-8dda894d9d81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-0aed62e2-kksd4\" (UID: \"b8c08b85-19ce-404f-b630-8dda894d9d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aed62e2-kksd4" Nov 01 06:31:44.726887 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:44.726810 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq7vl\" (UniqueName: \"kubernetes.io/projected/b8c08b85-19ce-404f-b630-8dda894d9d81-kube-api-access-tq7vl\") pod \"maas-api-revoke-keys-e2e-derive-0aed62e2-kksd4\" (UID: \"b8c08b85-19ce-404f-b630-8dda894d9d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aed62e2-kksd4" Nov 01 06:31:44.803417 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:44.803381 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aed62e2-kksd4" Nov 01 06:31:45.179115 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:45.179085 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aed62e2-kksd4"] Nov 01 06:31:45.179930 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:31:45.179902 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c08b85_19ce_404f_b630_8dda894d9d81.slice/crio-985a9a33eafd37a37252d3536ae315110b2b92f24a8d2dddebc652112590b3bd WatchSource:0}: Error finding container 985a9a33eafd37a37252d3536ae315110b2b92f24a8d2dddebc652112590b3bd: Status 404 returned error can't find the container with id 985a9a33eafd37a37252d3536ae315110b2b92f24a8d2dddebc652112590b3bd Nov 01 06:31:45.455365 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:45.455272 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aed62e2-kksd4" event={"ID":"b8c08b85-19ce-404f-b630-8dda894d9d81","Type":"ContainerStarted","Data":"985a9a33eafd37a37252d3536ae315110b2b92f24a8d2dddebc652112590b3bd"} Nov 01 06:31:46.464533 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:46.464411 5040 generic.go:356] "Generic (PLEG): container finished" podID="b8c08b85-19ce-404f-b630-8dda894d9d81" containerID="9ac2d9b451bf6bb4075fbe6ee9422d3bcd167b69c31a47e185095754297d74e8" exitCode=0 Nov 01 06:31:46.464533 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:46.464457 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aed62e2-kksd4" event={"ID":"b8c08b85-19ce-404f-b630-8dda894d9d81","Type":"ContainerDied","Data":"9ac2d9b451bf6bb4075fbe6ee9422d3bcd167b69c31a47e185095754297d74e8"} Nov 01 06:31:47.697447 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:47.697416 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aed62e2-kksd4" Nov 01 06:31:47.808359 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:47.808323 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8c08b85-19ce-404f-b630-8dda894d9d81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c08b85-19ce-404f-b630-8dda894d9d81-maas-api-xks-ca\") pod \"b8c08b85-19ce-404f-b630-8dda894d9d81\" (UID: \"b8c08b85-19ce-404f-b630-8dda894d9d81\") " Nov 01 06:31:47.808578 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:47.808450 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8c08b85-19ce-404f-b630-8dda894d9d81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c08b85-19ce-404f-b630-8dda894d9d81-maas-api-openshift-ca\") pod \"b8c08b85-19ce-404f-b630-8dda894d9d81\" (UID: \"b8c08b85-19ce-404f-b630-8dda894d9d81\") " Nov 01 06:31:47.808578 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:47.808552 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8c08b85-19ce-404f-b630-8dda894d9d81-kube-api-access-tq7vl\" (UniqueName: \"kubernetes.io/projected/b8c08b85-19ce-404f-b630-8dda894d9d81-kube-api-access-tq7vl\") pod \"b8c08b85-19ce-404f-b630-8dda894d9d81\" (UID: \"b8c08b85-19ce-404f-b630-8dda894d9d81\") " Nov 01 06:31:47.808916 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:47.808882 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8c08b85-19ce-404f-b630-8dda894d9d81-maas-api-openshift-ca" pod "b8c08b85-19ce-404f-b630-8dda894d9d81" (UID: "b8c08b85-19ce-404f-b630-8dda894d9d81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:31:47.810757 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:47.810727 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c08b85-19ce-404f-b630-8dda894d9d81-kube-api-access-tq7vl" pod "b8c08b85-19ce-404f-b630-8dda894d9d81" (UID: "b8c08b85-19ce-404f-b630-8dda894d9d81"). InnerVolumeSpecName "kube-api-access-tq7vl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:31:47.811018 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:47.810994 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8c08b85-19ce-404f-b630-8dda894d9d81-maas-api-xks-ca" pod "b8c08b85-19ce-404f-b630-8dda894d9d81" (UID: "b8c08b85-19ce-404f-b630-8dda894d9d81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:31:47.856086 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:47.856060 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:31:47.909780 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:47.909750 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq7vl\" (UniqueName: \"kubernetes.io/projected/b8c08b85-19ce-404f-b630-8dda894d9d81-kube-api-access-tq7vl\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:47.909780 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:47.909776 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c08b85-19ce-404f-b630-8dda894d9d81-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:47.909780 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:47.909787 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c08b85-19ce-404f-b630-8dda894d9d81-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:48.010751 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.010716 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/068716fa-4615-4dbf-9c5f-41687865d5b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/068716fa-4615-4dbf-9c5f-41687865d5b1-istio-envoy\") pod \"068716fa-4615-4dbf-9c5f-41687865d5b1\" (UID: \"068716fa-4615-4dbf-9c5f-41687865d5b1\") " Nov 01 06:31:48.010936 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.010767 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/068716fa-4615-4dbf-9c5f-41687865d5b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/068716fa-4615-4dbf-9c5f-41687865d5b1-workload-socket\") pod \"068716fa-4615-4dbf-9c5f-41687865d5b1\" (UID: \"068716fa-4615-4dbf-9c5f-41687865d5b1\") " Nov 01 06:31:48.010936 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.010824 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/068716fa-4615-4dbf-9c5f-41687865d5b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/068716fa-4615-4dbf-9c5f-41687865d5b1-istio-podinfo\") pod \"068716fa-4615-4dbf-9c5f-41687865d5b1\" (UID: \"068716fa-4615-4dbf-9c5f-41687865d5b1\") " Nov 01 06:31:48.010936 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.010862 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/068716fa-4615-4dbf-9c5f-41687865d5b1-kube-api-access-l9btp\" (UniqueName: \"kubernetes.io/projected/068716fa-4615-4dbf-9c5f-41687865d5b1-kube-api-access-l9btp\") pod \"068716fa-4615-4dbf-9c5f-41687865d5b1\" (UID: \"068716fa-4615-4dbf-9c5f-41687865d5b1\") " Nov 01 06:31:48.010936 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.010891 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/068716fa-4615-4dbf-9c5f-41687865d5b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/068716fa-4615-4dbf-9c5f-41687865d5b1-workload-certs\") pod \"068716fa-4615-4dbf-9c5f-41687865d5b1\" (UID: \"068716fa-4615-4dbf-9c5f-41687865d5b1\") " Nov 01 06:31:48.010936 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.010919 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/068716fa-4615-4dbf-9c5f-41687865d5b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/068716fa-4615-4dbf-9c5f-41687865d5b1-credential-socket\") pod \"068716fa-4615-4dbf-9c5f-41687865d5b1\" (UID: \"068716fa-4615-4dbf-9c5f-41687865d5b1\") " Nov 01 06:31:48.011231 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.010959 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/068716fa-4615-4dbf-9c5f-41687865d5b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/068716fa-4615-4dbf-9c5f-41687865d5b1-istio-data\") pod \"068716fa-4615-4dbf-9c5f-41687865d5b1\" (UID: \"068716fa-4615-4dbf-9c5f-41687865d5b1\") " Nov 01 06:31:48.011231 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.011108 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/068716fa-4615-4dbf-9c5f-41687865d5b1-istio-token\" (UniqueName: \"kubernetes.io/projected/068716fa-4615-4dbf-9c5f-41687865d5b1-istio-token\") pod \"068716fa-4615-4dbf-9c5f-41687865d5b1\" (UID: \"068716fa-4615-4dbf-9c5f-41687865d5b1\") " Nov 01 06:31:48.011231 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.011113 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068716fa-4615-4dbf-9c5f-41687865d5b1-workload-socket" pod "068716fa-4615-4dbf-9c5f-41687865d5b1" (UID: "068716fa-4615-4dbf-9c5f-41687865d5b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:31:48.011414 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.011231 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/068716fa-4615-4dbf-9c5f-41687865d5b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/068716fa-4615-4dbf-9c5f-41687865d5b1-istiod-ca-cert\") pod \"068716fa-4615-4dbf-9c5f-41687865d5b1\" (UID: \"068716fa-4615-4dbf-9c5f-41687865d5b1\") " Nov 01 06:31:48.011560 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.011533 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068716fa-4615-4dbf-9c5f-41687865d5b1-credential-socket" pod "068716fa-4615-4dbf-9c5f-41687865d5b1" (UID: "068716fa-4615-4dbf-9c5f-41687865d5b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:31:48.011652 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.011608 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068716fa-4615-4dbf-9c5f-41687865d5b1-istio-data" pod "068716fa-4615-4dbf-9c5f-41687865d5b1" (UID: "068716fa-4615-4dbf-9c5f-41687865d5b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:31:48.011812 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.011768 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/068716fa-4615-4dbf-9c5f-41687865d5b1-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:48.011911 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.011811 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/068716fa-4615-4dbf-9c5f-41687865d5b1-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:48.011911 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.011828 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/068716fa-4615-4dbf-9c5f-41687865d5b1-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:48.012175 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.012140 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/068716fa-4615-4dbf-9c5f-41687865d5b1-istiod-ca-cert" pod "068716fa-4615-4dbf-9c5f-41687865d5b1" (UID: "068716fa-4615-4dbf-9c5f-41687865d5b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:31:48.012430 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.012400 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068716fa-4615-4dbf-9c5f-41687865d5b1-workload-certs" pod "068716fa-4615-4dbf-9c5f-41687865d5b1" (UID: "068716fa-4615-4dbf-9c5f-41687865d5b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:31:48.013360 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.013333 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/068716fa-4615-4dbf-9c5f-41687865d5b1-istio-token" pod "068716fa-4615-4dbf-9c5f-41687865d5b1" (UID: "068716fa-4615-4dbf-9c5f-41687865d5b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:31:48.013458 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.013394 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/068716fa-4615-4dbf-9c5f-41687865d5b1-istio-podinfo" pod "068716fa-4615-4dbf-9c5f-41687865d5b1" (UID: "068716fa-4615-4dbf-9c5f-41687865d5b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:31:48.013546 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.013529 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068716fa-4615-4dbf-9c5f-41687865d5b1-istio-envoy" pod "068716fa-4615-4dbf-9c5f-41687865d5b1" (UID: "068716fa-4615-4dbf-9c5f-41687865d5b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:31:48.014151 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.014134 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/068716fa-4615-4dbf-9c5f-41687865d5b1-kube-api-access-l9btp" pod "068716fa-4615-4dbf-9c5f-41687865d5b1" (UID: "068716fa-4615-4dbf-9c5f-41687865d5b1"). InnerVolumeSpecName "kube-api-access-l9btp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:31:48.113340 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.113257 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/068716fa-4615-4dbf-9c5f-41687865d5b1-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:48.113340 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.113287 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/068716fa-4615-4dbf-9c5f-41687865d5b1-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:48.113340 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.113297 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/068716fa-4615-4dbf-9c5f-41687865d5b1-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:48.113340 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.113306 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/068716fa-4615-4dbf-9c5f-41687865d5b1-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:48.113340 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.113316 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9btp\" (UniqueName: \"kubernetes.io/projected/068716fa-4615-4dbf-9c5f-41687865d5b1-kube-api-access-l9btp\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:48.113340 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.113327 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/068716fa-4615-4dbf-9c5f-41687865d5b1-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:48.481506 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.481406 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aed62e2-kksd4" Nov 01 06:31:48.481506 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.481447 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aed62e2-kksd4" event={"ID":"b8c08b85-19ce-404f-b630-8dda894d9d81","Type":"ContainerDied","Data":"985a9a33eafd37a37252d3536ae315110b2b92f24a8d2dddebc652112590b3bd"} Nov 01 06:31:48.481717 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.481529 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="985a9a33eafd37a37252d3536ae315110b2b92f24a8d2dddebc652112590b3bd" Nov 01 06:31:48.483470 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.483446 5040 generic.go:356] "Generic (PLEG): container finished" podID="068716fa-4615-4dbf-9c5f-41687865d5b1" containerID="3b141c3d9dc299bf597a555cd2cfdc2c9af0a7d6b9a3b4d1970e729214da666d" exitCode=0 Nov 01 06:31:48.483629 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.483515 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" event={"ID":"068716fa-4615-4dbf-9c5f-41687865d5b1","Type":"ContainerDied","Data":"3b141c3d9dc299bf597a555cd2cfdc2c9af0a7d6b9a3b4d1970e729214da666d"} Nov 01 06:31:48.483629 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.483528 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" Nov 01 06:31:48.483629 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.483567 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt" event={"ID":"068716fa-4615-4dbf-9c5f-41687865d5b1","Type":"ContainerDied","Data":"35fc601a2ab248de93b730888250e9bf7c0dc6e21e97f2d146b52dac21b23645"} Nov 01 06:31:48.483629 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.483597 5040 scope.go:122] "RemoveContainer" containerID="3b141c3d9dc299bf597a555cd2cfdc2c9af0a7d6b9a3b4d1970e729214da666d" Nov 01 06:31:48.495584 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.495559 5040 scope.go:122] "RemoveContainer" containerID="44791eecd8e3223d742ac4c5c0b2f07057100ca4306ee7c0a39e839394079a15" Nov 01 06:31:48.511686 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.511658 5040 scope.go:122] "RemoveContainer" containerID="3b141c3d9dc299bf597a555cd2cfdc2c9af0a7d6b9a3b4d1970e729214da666d" Nov 01 06:31:48.512019 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:31:48.511999 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b141c3d9dc299bf597a555cd2cfdc2c9af0a7d6b9a3b4d1970e729214da666d\": container with ID starting with 3b141c3d9dc299bf597a555cd2cfdc2c9af0a7d6b9a3b4d1970e729214da666d not found: ID does not exist" containerID="3b141c3d9dc299bf597a555cd2cfdc2c9af0a7d6b9a3b4d1970e729214da666d" Nov 01 06:31:48.512082 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.512027 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b141c3d9dc299bf597a555cd2cfdc2c9af0a7d6b9a3b4d1970e729214da666d"} err="failed to get container status \"3b141c3d9dc299bf597a555cd2cfdc2c9af0a7d6b9a3b4d1970e729214da666d\": rpc error: code = NotFound desc = could not find container \"3b141c3d9dc299bf597a555cd2cfdc2c9af0a7d6b9a3b4d1970e729214da666d\": container with ID starting with 3b141c3d9dc299bf597a555cd2cfdc2c9af0a7d6b9a3b4d1970e729214da666d not found: ID does not exist" Nov 01 06:31:48.512082 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.512041 5040 scope.go:122] "RemoveContainer" containerID="44791eecd8e3223d742ac4c5c0b2f07057100ca4306ee7c0a39e839394079a15" Nov 01 06:31:48.512296 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:31:48.512281 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44791eecd8e3223d742ac4c5c0b2f07057100ca4306ee7c0a39e839394079a15\": container with ID starting with 44791eecd8e3223d742ac4c5c0b2f07057100ca4306ee7c0a39e839394079a15 not found: ID does not exist" containerID="44791eecd8e3223d742ac4c5c0b2f07057100ca4306ee7c0a39e839394079a15" Nov 01 06:31:48.512340 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.512299 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44791eecd8e3223d742ac4c5c0b2f07057100ca4306ee7c0a39e839394079a15"} err="failed to get container status \"44791eecd8e3223d742ac4c5c0b2f07057100ca4306ee7c0a39e839394079a15\": rpc error: code = NotFound desc = could not find container \"44791eecd8e3223d742ac4c5c0b2f07057100ca4306ee7c0a39e839394079a15\": container with ID starting with 44791eecd8e3223d742ac4c5c0b2f07057100ca4306ee7c0a39e839394079a15 not found: ID does not exist" Nov 01 06:31:48.704086 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.704052 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt"] Nov 01 06:31:48.754768 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:48.754737 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f13905-openshift-default-6b76f49cb9-4k9rt"] Nov 01 06:31:49.515011 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:31:49.514982 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="068716fa-4615-4dbf-9c5f-41687865d5b1" path="/var/lib/kubelet/pods/068716fa-4615-4dbf-9c5f-41687865d5b1/volumes" Nov 01 06:32:03.845523 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:03.845471 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk"] Nov 01 06:32:03.846069 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:03.845710 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" podUID="f2e934b9-b834-487b-9a7b-08f3047c49c6" containerName="istio-proxy" containerID="cri-o://0b0140b0931a87b10ae9297a6e3f35854cbe6c0a697e52f9a8d4c8e9d9249909" gracePeriod=30 Nov 01 06:32:09.110806 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.110749 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:32:09.185506 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.185442 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2e934b9-b834-487b-9a7b-08f3047c49c6-istio-token\" (UniqueName: \"kubernetes.io/projected/f2e934b9-b834-487b-9a7b-08f3047c49c6-istio-token\") pod \"f2e934b9-b834-487b-9a7b-08f3047c49c6\" (UID: \"f2e934b9-b834-487b-9a7b-08f3047c49c6\") " Nov 01 06:32:09.185685 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.185578 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2e934b9-b834-487b-9a7b-08f3047c49c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2e934b9-b834-487b-9a7b-08f3047c49c6-istiod-ca-cert\") pod \"f2e934b9-b834-487b-9a7b-08f3047c49c6\" (UID: \"f2e934b9-b834-487b-9a7b-08f3047c49c6\") " Nov 01 06:32:09.185685 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.185632 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2e934b9-b834-487b-9a7b-08f3047c49c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2e934b9-b834-487b-9a7b-08f3047c49c6-istio-data\") pod \"f2e934b9-b834-487b-9a7b-08f3047c49c6\" (UID: \"f2e934b9-b834-487b-9a7b-08f3047c49c6\") " Nov 01 06:32:09.185685 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.185673 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2e934b9-b834-487b-9a7b-08f3047c49c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2e934b9-b834-487b-9a7b-08f3047c49c6-credential-socket\") pod \"f2e934b9-b834-487b-9a7b-08f3047c49c6\" (UID: \"f2e934b9-b834-487b-9a7b-08f3047c49c6\") " Nov 01 06:32:09.185882 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.185715 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2e934b9-b834-487b-9a7b-08f3047c49c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2e934b9-b834-487b-9a7b-08f3047c49c6-workload-certs\") pod \"f2e934b9-b834-487b-9a7b-08f3047c49c6\" (UID: \"f2e934b9-b834-487b-9a7b-08f3047c49c6\") " Nov 01 06:32:09.185882 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.185766 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2e934b9-b834-487b-9a7b-08f3047c49c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2e934b9-b834-487b-9a7b-08f3047c49c6-workload-socket\") pod \"f2e934b9-b834-487b-9a7b-08f3047c49c6\" (UID: \"f2e934b9-b834-487b-9a7b-08f3047c49c6\") " Nov 01 06:32:09.185882 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.185801 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2e934b9-b834-487b-9a7b-08f3047c49c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2e934b9-b834-487b-9a7b-08f3047c49c6-istio-envoy\") pod \"f2e934b9-b834-487b-9a7b-08f3047c49c6\" (UID: \"f2e934b9-b834-487b-9a7b-08f3047c49c6\") " Nov 01 06:32:09.185882 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.185869 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2e934b9-b834-487b-9a7b-08f3047c49c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2e934b9-b834-487b-9a7b-08f3047c49c6-istio-podinfo\") pod \"f2e934b9-b834-487b-9a7b-08f3047c49c6\" (UID: \"f2e934b9-b834-487b-9a7b-08f3047c49c6\") " Nov 01 06:32:09.186097 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.185922 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2e934b9-b834-487b-9a7b-08f3047c49c6-kube-api-access-p24qh\" (UniqueName: \"kubernetes.io/projected/f2e934b9-b834-487b-9a7b-08f3047c49c6-kube-api-access-p24qh\") pod \"f2e934b9-b834-487b-9a7b-08f3047c49c6\" (UID: \"f2e934b9-b834-487b-9a7b-08f3047c49c6\") " Nov 01 06:32:09.186097 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.185983 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e934b9-b834-487b-9a7b-08f3047c49c6-credential-socket" pod "f2e934b9-b834-487b-9a7b-08f3047c49c6" (UID: "f2e934b9-b834-487b-9a7b-08f3047c49c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:32:09.186097 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.186007 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2e934b9-b834-487b-9a7b-08f3047c49c6-istiod-ca-cert" pod "f2e934b9-b834-487b-9a7b-08f3047c49c6" (UID: "f2e934b9-b834-487b-9a7b-08f3047c49c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:32:09.186097 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.186009 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e934b9-b834-487b-9a7b-08f3047c49c6-workload-certs" pod "f2e934b9-b834-487b-9a7b-08f3047c49c6" (UID: "f2e934b9-b834-487b-9a7b-08f3047c49c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:32:09.186097 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.186067 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e934b9-b834-487b-9a7b-08f3047c49c6-workload-socket" pod "f2e934b9-b834-487b-9a7b-08f3047c49c6" (UID: "f2e934b9-b834-487b-9a7b-08f3047c49c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:32:09.186382 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.186297 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e934b9-b834-487b-9a7b-08f3047c49c6-istio-data" pod "f2e934b9-b834-487b-9a7b-08f3047c49c6" (UID: "f2e934b9-b834-487b-9a7b-08f3047c49c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:32:09.186450 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.186431 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2e934b9-b834-487b-9a7b-08f3047c49c6-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:32:09.186525 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.186455 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2e934b9-b834-487b-9a7b-08f3047c49c6-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:32:09.186525 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.186470 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2e934b9-b834-487b-9a7b-08f3047c49c6-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:32:09.186525 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.186502 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2e934b9-b834-487b-9a7b-08f3047c49c6-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:32:09.186525 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.186515 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2e934b9-b834-487b-9a7b-08f3047c49c6-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:32:09.188003 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.187980 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e934b9-b834-487b-9a7b-08f3047c49c6-istio-envoy" pod "f2e934b9-b834-487b-9a7b-08f3047c49c6" (UID: "f2e934b9-b834-487b-9a7b-08f3047c49c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:32:09.188110 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.188019 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2e934b9-b834-487b-9a7b-08f3047c49c6-kube-api-access-p24qh" pod "f2e934b9-b834-487b-9a7b-08f3047c49c6" (UID: "f2e934b9-b834-487b-9a7b-08f3047c49c6"). InnerVolumeSpecName "kube-api-access-p24qh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:32:09.188168 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.188106 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2e934b9-b834-487b-9a7b-08f3047c49c6-istio-token" pod "f2e934b9-b834-487b-9a7b-08f3047c49c6" (UID: "f2e934b9-b834-487b-9a7b-08f3047c49c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:32:09.188244 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.188197 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2e934b9-b834-487b-9a7b-08f3047c49c6-istio-podinfo" pod "f2e934b9-b834-487b-9a7b-08f3047c49c6" (UID: "f2e934b9-b834-487b-9a7b-08f3047c49c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:32:09.288076 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.288041 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2e934b9-b834-487b-9a7b-08f3047c49c6-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:32:09.288352 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.288297 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2e934b9-b834-487b-9a7b-08f3047c49c6-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:32:09.288352 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.288320 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2e934b9-b834-487b-9a7b-08f3047c49c6-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:32:09.288352 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.288333 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p24qh\" (UniqueName: \"kubernetes.io/projected/f2e934b9-b834-487b-9a7b-08f3047c49c6-kube-api-access-p24qh\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:32:09.670038 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.669998 5040 generic.go:356] "Generic (PLEG): container finished" podID="f2e934b9-b834-487b-9a7b-08f3047c49c6" containerID="0b0140b0931a87b10ae9297a6e3f35854cbe6c0a697e52f9a8d4c8e9d9249909" exitCode=0 Nov 01 06:32:09.670363 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.670339 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" event={"ID":"f2e934b9-b834-487b-9a7b-08f3047c49c6","Type":"ContainerDied","Data":"0b0140b0931a87b10ae9297a6e3f35854cbe6c0a697e52f9a8d4c8e9d9249909"} Nov 01 06:32:09.670610 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.670592 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" event={"ID":"f2e934b9-b834-487b-9a7b-08f3047c49c6","Type":"ContainerDied","Data":"c185d891fff98e841d23289b330b89ec62882f4a058a6e42b1e235f5e050a265"} Nov 01 06:32:09.670801 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.670786 5040 scope.go:122] "RemoveContainer" containerID="0b0140b0931a87b10ae9297a6e3f35854cbe6c0a697e52f9a8d4c8e9d9249909" Nov 01 06:32:09.671199 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.671107 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk" Nov 01 06:32:09.683467 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.683444 5040 scope.go:122] "RemoveContainer" containerID="2dd9ae47500e93ec5adb451632795064e3ce242c61f8461f6f04a6931751c19d" Nov 01 06:32:09.693043 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.693023 5040 scope.go:122] "RemoveContainer" containerID="0b0140b0931a87b10ae9297a6e3f35854cbe6c0a697e52f9a8d4c8e9d9249909" Nov 01 06:32:09.693340 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:32:09.693318 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b0140b0931a87b10ae9297a6e3f35854cbe6c0a697e52f9a8d4c8e9d9249909\": container with ID starting with 0b0140b0931a87b10ae9297a6e3f35854cbe6c0a697e52f9a8d4c8e9d9249909 not found: ID does not exist" containerID="0b0140b0931a87b10ae9297a6e3f35854cbe6c0a697e52f9a8d4c8e9d9249909" Nov 01 06:32:09.693387 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.693351 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b0140b0931a87b10ae9297a6e3f35854cbe6c0a697e52f9a8d4c8e9d9249909"} err="failed to get container status \"0b0140b0931a87b10ae9297a6e3f35854cbe6c0a697e52f9a8d4c8e9d9249909\": rpc error: code = NotFound desc = could not find container \"0b0140b0931a87b10ae9297a6e3f35854cbe6c0a697e52f9a8d4c8e9d9249909\": container with ID starting with 0b0140b0931a87b10ae9297a6e3f35854cbe6c0a697e52f9a8d4c8e9d9249909 not found: ID does not exist" Nov 01 06:32:09.693387 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.693373 5040 scope.go:122] "RemoveContainer" containerID="2dd9ae47500e93ec5adb451632795064e3ce242c61f8461f6f04a6931751c19d" Nov 01 06:32:09.693692 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:32:09.693671 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dd9ae47500e93ec5adb451632795064e3ce242c61f8461f6f04a6931751c19d\": container with ID starting with 2dd9ae47500e93ec5adb451632795064e3ce242c61f8461f6f04a6931751c19d not found: ID does not exist" containerID="2dd9ae47500e93ec5adb451632795064e3ce242c61f8461f6f04a6931751c19d" Nov 01 06:32:09.693752 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.693700 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd9ae47500e93ec5adb451632795064e3ce242c61f8461f6f04a6931751c19d"} err="failed to get container status \"2dd9ae47500e93ec5adb451632795064e3ce242c61f8461f6f04a6931751c19d\": rpc error: code = NotFound desc = could not find container \"2dd9ae47500e93ec5adb451632795064e3ce242c61f8461f6f04a6931751c19d\": container with ID starting with 2dd9ae47500e93ec5adb451632795064e3ce242c61f8461f6f04a6931751c19d not found: ID does not exist" Nov 01 06:32:09.840865 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.840783 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk"] Nov 01 06:32:09.843978 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:09.843956 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-745a18-openshift-default-9d949867f-kq5zk"] Nov 01 06:32:09.972970 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:32:09.972930 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:32:11.515898 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:11.515868 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2e934b9-b834-487b-9a7b-08f3047c49c6" path="/var/lib/kubelet/pods/f2e934b9-b834-487b-9a7b-08f3047c49c6/volumes" Nov 01 06:32:15.273829 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.273794 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82"] Nov 01 06:32:15.274322 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.274302 5040 cpu_manager.go:414] "containerMap: removing container" podUID="068716fa-4615-4dbf-9c5f-41687865d5b1" containerName="istio-proxy" Nov 01 06:32:15.274368 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.274326 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="068716fa-4615-4dbf-9c5f-41687865d5b1" containerName="istio-proxy" Nov 01 06:32:15.274368 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.274351 5040 cpu_manager.go:414] "containerMap: removing container" podUID="f2e934b9-b834-487b-9a7b-08f3047c49c6" containerName="istio-proxy" Nov 01 06:32:15.274368 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.274358 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2e934b9-b834-487b-9a7b-08f3047c49c6" containerName="istio-proxy" Nov 01 06:32:15.274475 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.274372 5040 cpu_manager.go:414] "containerMap: removing container" podUID="f2e934b9-b834-487b-9a7b-08f3047c49c6" containerName="istio-proxy" Nov 01 06:32:15.274475 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.274379 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2e934b9-b834-487b-9a7b-08f3047c49c6" containerName="istio-proxy" Nov 01 06:32:15.274475 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.274408 5040 cpu_manager.go:414] "containerMap: removing container" podUID="b8c08b85-19ce-404f-b630-8dda894d9d81" containerName="revoke-keys" Nov 01 06:32:15.274475 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.274415 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8c08b85-19ce-404f-b630-8dda894d9d81" containerName="revoke-keys" Nov 01 06:32:15.274626 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.274581 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="068716fa-4615-4dbf-9c5f-41687865d5b1" containerName="istio-proxy" Nov 01 06:32:15.274626 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.274594 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2e934b9-b834-487b-9a7b-08f3047c49c6" containerName="istio-proxy" Nov 01 06:32:15.274626 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.274604 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2e934b9-b834-487b-9a7b-08f3047c49c6" containerName="istio-proxy" Nov 01 06:32:15.274626 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.274617 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8c08b85-19ce-404f-b630-8dda894d9d81" containerName="revoke-keys" Nov 01 06:32:15.279850 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.279826 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:32:15.333077 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.333045 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-3680c5-openshift-default-dockercfg-vxmjs\"" Nov 01 06:32:15.373502 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.368945 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9577376-97a1-4915-b69e-26e60c29fce4-istio-token\") pod \"e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82\" (UID: \"d9577376-97a1-4915-b69e-26e60c29fce4\") " pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:32:15.373502 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.369023 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9577376-97a1-4915-b69e-26e60c29fce4-istio-podinfo\") pod \"e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82\" (UID: \"d9577376-97a1-4915-b69e-26e60c29fce4\") " pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:32:15.373502 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.369047 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9577376-97a1-4915-b69e-26e60c29fce4-istiod-ca-cert\") pod \"e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82\" (UID: \"d9577376-97a1-4915-b69e-26e60c29fce4\") " pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:32:15.373502 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.369085 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9577376-97a1-4915-b69e-26e60c29fce4-credential-socket\") pod \"e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82\" (UID: \"d9577376-97a1-4915-b69e-26e60c29fce4\") " pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:32:15.373502 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.369102 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9577376-97a1-4915-b69e-26e60c29fce4-istio-envoy\") pod \"e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82\" (UID: \"d9577376-97a1-4915-b69e-26e60c29fce4\") " pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:32:15.373502 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.369156 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9vnb\" (UniqueName: \"kubernetes.io/projected/d9577376-97a1-4915-b69e-26e60c29fce4-kube-api-access-j9vnb\") pod \"e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82\" (UID: \"d9577376-97a1-4915-b69e-26e60c29fce4\") " pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:32:15.373502 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.369180 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9577376-97a1-4915-b69e-26e60c29fce4-workload-certs\") pod \"e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82\" (UID: \"d9577376-97a1-4915-b69e-26e60c29fce4\") " pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:32:15.373502 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.369218 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9577376-97a1-4915-b69e-26e60c29fce4-istio-data\") pod \"e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82\" (UID: \"d9577376-97a1-4915-b69e-26e60c29fce4\") " pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:32:15.373502 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.369259 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9577376-97a1-4915-b69e-26e60c29fce4-workload-socket\") pod \"e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82\" (UID: \"d9577376-97a1-4915-b69e-26e60c29fce4\") " pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:32:15.428579 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.428537 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82"] Nov 01 06:32:15.471246 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.471204 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9577376-97a1-4915-b69e-26e60c29fce4-credential-socket\") pod \"e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82\" (UID: \"d9577376-97a1-4915-b69e-26e60c29fce4\") " pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:32:15.472313 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.471932 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9577376-97a1-4915-b69e-26e60c29fce4-istio-envoy\") pod \"e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82\" (UID: \"d9577376-97a1-4915-b69e-26e60c29fce4\") " pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:32:15.483664 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.471857 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9577376-97a1-4915-b69e-26e60c29fce4-credential-socket\") pod \"e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82\" (UID: \"d9577376-97a1-4915-b69e-26e60c29fce4\") " pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:32:15.483664 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.475331 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9577376-97a1-4915-b69e-26e60c29fce4-istio-envoy\") pod \"e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82\" (UID: \"d9577376-97a1-4915-b69e-26e60c29fce4\") " pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:32:15.484430 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.484006 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9vnb\" (UniqueName: \"kubernetes.io/projected/d9577376-97a1-4915-b69e-26e60c29fce4-kube-api-access-j9vnb\") pod \"e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82\" (UID: \"d9577376-97a1-4915-b69e-26e60c29fce4\") " pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:32:15.488346 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.488297 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9577376-97a1-4915-b69e-26e60c29fce4-workload-certs\") pod \"e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82\" (UID: \"d9577376-97a1-4915-b69e-26e60c29fce4\") " pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:32:15.488586 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.488531 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9577376-97a1-4915-b69e-26e60c29fce4-workload-certs\") pod \"e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82\" (UID: \"d9577376-97a1-4915-b69e-26e60c29fce4\") " pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:32:15.488776 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.488737 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9577376-97a1-4915-b69e-26e60c29fce4-istio-data\") pod \"e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82\" (UID: \"d9577376-97a1-4915-b69e-26e60c29fce4\") " pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:32:15.489201 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.489137 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9577376-97a1-4915-b69e-26e60c29fce4-istio-data\") pod \"e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82\" (UID: \"d9577376-97a1-4915-b69e-26e60c29fce4\") " pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:32:15.489488 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.489403 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9577376-97a1-4915-b69e-26e60c29fce4-workload-socket\") pod \"e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82\" (UID: \"d9577376-97a1-4915-b69e-26e60c29fce4\") " pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:32:15.492518 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.490860 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9577376-97a1-4915-b69e-26e60c29fce4-workload-socket\") pod \"e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82\" (UID: \"d9577376-97a1-4915-b69e-26e60c29fce4\") " pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:32:15.492809 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.492754 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9577376-97a1-4915-b69e-26e60c29fce4-istio-token\") pod \"e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82\" (UID: \"d9577376-97a1-4915-b69e-26e60c29fce4\") " pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:32:15.493074 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.493010 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9577376-97a1-4915-b69e-26e60c29fce4-istio-podinfo\") pod \"e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82\" (UID: \"d9577376-97a1-4915-b69e-26e60c29fce4\") " pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:32:15.493252 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.493051 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9577376-97a1-4915-b69e-26e60c29fce4-istiod-ca-cert\") pod \"e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82\" (UID: \"d9577376-97a1-4915-b69e-26e60c29fce4\") " pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:32:15.496257 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.493933 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9577376-97a1-4915-b69e-26e60c29fce4-istiod-ca-cert\") pod \"e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82\" (UID: \"d9577376-97a1-4915-b69e-26e60c29fce4\") " pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:32:15.510608 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.510577 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9577376-97a1-4915-b69e-26e60c29fce4-istio-podinfo\") pod \"e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82\" (UID: \"d9577376-97a1-4915-b69e-26e60c29fce4\") " pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:32:15.546228 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.546145 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9vnb\" (UniqueName: \"kubernetes.io/projected/d9577376-97a1-4915-b69e-26e60c29fce4-kube-api-access-j9vnb\") pod \"e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82\" (UID: \"d9577376-97a1-4915-b69e-26e60c29fce4\") " pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:32:15.557807 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.557774 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9577376-97a1-4915-b69e-26e60c29fce4-istio-token\") pod \"e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82\" (UID: \"d9577376-97a1-4915-b69e-26e60c29fce4\") " pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:32:15.595140 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:15.595110 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:32:16.034210 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:16.034183 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82"] Nov 01 06:32:16.035312 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:32:16.035280 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9577376_97a1_4915_b69e_26e60c29fce4.slice/crio-5b4dd48d84d752b362f13b97687af08cee0adf64c43d9b80dcb70122b78915e5 WatchSource:0}: Error finding container 5b4dd48d84d752b362f13b97687af08cee0adf64c43d9b80dcb70122b78915e5: Status 404 returned error can't find the container with id 5b4dd48d84d752b362f13b97687af08cee0adf64c43d9b80dcb70122b78915e5 Nov 01 06:32:16.037846 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:16.037812 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:32:16.037934 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:16.037882 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:32:16.037934 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:16.037909 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:32:16.731549 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:16.731516 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" event={"ID":"d9577376-97a1-4915-b69e-26e60c29fce4","Type":"ContainerStarted","Data":"3dfe98b97d63aa55aa6310dee9141b058482d425f4b7115638bbf0e0bd5ea214"} Nov 01 06:32:16.731952 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:16.731574 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" event={"ID":"d9577376-97a1-4915-b69e-26e60c29fce4","Type":"ContainerStarted","Data":"5b4dd48d84d752b362f13b97687af08cee0adf64c43d9b80dcb70122b78915e5"} Nov 01 06:32:16.855846 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:16.855794 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" podStartSLOduration=1.855781788 podStartE2EDuration="1.855781788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:32:16.853380121 +0000 UTC m=+2815.508497289" watchObservedRunningTime="2026-11-01 06:32:16.855781788 +0000 UTC m=+2815.510898955" Nov 01 06:32:17.595946 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:17.595914 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:32:17.601971 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:17.601945 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:32:17.738334 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:17.738297 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:32:17.739265 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:17.739245 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:32:23.175953 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:23.175922 5040 scope.go:122] "RemoveContainer" containerID="9aa305c39370e675c2ba2589af43f496757d652419e799bd4dc51405761ccfae" Nov 01 06:32:23.185918 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:23.185897 5040 scope.go:122] "RemoveContainer" containerID="461836d3e61393cbe7bb9e76ee1770b28d4c570f34509590e092bd9cdc7b25ce" Nov 01 06:32:23.195646 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:23.195623 5040 scope.go:122] "RemoveContainer" containerID="573eb279e5a63355b8a3ea7c7e78b68429fb6b9b4a46f32031fb05c948e54db5" Nov 01 06:32:23.204944 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:23.204926 5040 scope.go:122] "RemoveContainer" containerID="2519dcc32bbaed2948fa886551cd783ac8b0158fb71a8e13ebb02b584972a952" Nov 01 06:32:23.213413 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:23.213388 5040 scope.go:122] "RemoveContainer" containerID="2332a69512af9dce998a62285a4765d60fabe5e67447e55cfbb4d9f965ce52a0" Nov 01 06:32:23.221944 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:23.221922 5040 scope.go:122] "RemoveContainer" containerID="7a35bf2e13324befe3d0e1691cb73dcc27ee2bb3728c4da31bcd6c584c427784" Nov 01 06:32:27.047636 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:27.047598 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a802d7ec-zrw5m"] Nov 01 06:32:27.080892 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:27.080860 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a802d7ec-zrw5m"] Nov 01 06:32:27.525915 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:27.525875 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cffa541-7c5c-41c9-8c14-4a97a9375234" path="/var/lib/kubelet/pods/5cffa541-7c5c-41c9-8c14-4a97a9375234/volumes" Nov 01 06:32:34.881602 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:34.881572 5040 generic.go:356] "Generic (PLEG): container finished" podID="14b4da92-5164-45df-8306-5f228a7c5ac2" containerID="ff3ac7610ee3e1a638c58bed2ebc6b8eff09ecc403f1b740651aba81e65c6228" exitCode=0 Nov 01 06:32:34.881997 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:34.881652 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" event={"ID":"14b4da92-5164-45df-8306-5f228a7c5ac2","Type":"ContainerDied","Data":"ff3ac7610ee3e1a638c58bed2ebc6b8eff09ecc403f1b740651aba81e65c6228"} Nov 01 06:32:34.882078 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:34.882064 5040 scope.go:122] "RemoveContainer" containerID="ff3ac7610ee3e1a638c58bed2ebc6b8eff09ecc403f1b740651aba81e65c6228" Nov 01 06:32:34.883601 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:34.883572 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:32:34.883707 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:34.883656 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:32:34.883772 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:34.883705 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:32:35.890606 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:35.890565 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" event={"ID":"14b4da92-5164-45df-8306-5f228a7c5ac2","Type":"ContainerStarted","Data":"61e42cad06057a6933769c825bb406fe1448b60b24169e0bcd8a699045509ed9"} Nov 01 06:32:36.788246 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:36.788207 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:32:36.788523 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:36.788473 5040 patch_prober.go:29] interesting pod/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 01 06:32:36.788579 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:36.788557 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" podUID="14b4da92-5164-45df-8306-5f228a7c5ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 01 06:32:37.788363 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:37.788325 5040 patch_prober.go:29] interesting pod/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 01 06:32:37.788830 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:37.788379 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" podUID="14b4da92-5164-45df-8306-5f228a7c5ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 01 06:32:38.788408 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:38.788369 5040 patch_prober.go:29] interesting pod/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 01 06:32:38.788849 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:38.788429 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" podUID="14b4da92-5164-45df-8306-5f228a7c5ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 01 06:32:39.788459 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:39.788423 5040 patch_prober.go:29] interesting pod/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 01 06:32:39.788894 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:39.788521 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" podUID="14b4da92-5164-45df-8306-5f228a7c5ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 01 06:32:39.973352 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:32:39.973315 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:32:40.788655 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:40.788617 5040 patch_prober.go:29] interesting pod/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 01 06:32:40.789088 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:40.788674 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" podUID="14b4da92-5164-45df-8306-5f228a7c5ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 01 06:32:41.794171 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:41.794136 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:32:41.794635 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:41.794522 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:32:41.795528 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:32:41.795504 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:33:05.179640 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:05.179607 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aed62e2-kksd4"] Nov 01 06:33:05.233023 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:05.232974 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aed62e2-kksd4"] Nov 01 06:33:05.515063 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:05.515033 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8c08b85-19ce-404f-b630-8dda894d9d81" path="/var/lib/kubelet/pods/b8c08b85-19ce-404f-b630-8dda894d9d81/volumes" Nov 01 06:33:09.972595 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:33:09.972553 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:33:15.485501 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:15.485450 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7134-95f4a7"] Nov 01 06:33:15.487126 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:15.487091 5040 cpu_manager.go:414] "containerMap: removing container" podUID="068716fa-4615-4dbf-9c5f-41687865d5b1" containerName="istio-proxy" Nov 01 06:33:15.487126 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:15.487125 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="068716fa-4615-4dbf-9c5f-41687865d5b1" containerName="istio-proxy" Nov 01 06:33:15.487473 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:15.487457 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="068716fa-4615-4dbf-9c5f-41687865d5b1" containerName="istio-proxy" Nov 01 06:33:15.492879 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:15.492857 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7134-95f4a7" Nov 01 06:33:15.539821 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:15.539774 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7134-95f4a7"] Nov 01 06:33:15.585726 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:15.585697 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hgn5l\"" Nov 01 06:33:15.630796 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:15.630760 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5p5n\" (UniqueName: \"kubernetes.io/projected/4bec169a-6a5f-4b30-bf2a-5a83f2fce624-kube-api-access-j5p5n\") pod \"test-curl-7134-95f4a7\" (UID: \"4bec169a-6a5f-4b30-bf2a-5a83f2fce624\") " pod="opendatahub/test-curl-7134-95f4a7" Nov 01 06:33:15.732288 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:15.732248 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5p5n\" (UniqueName: \"kubernetes.io/projected/4bec169a-6a5f-4b30-bf2a-5a83f2fce624-kube-api-access-j5p5n\") pod \"test-curl-7134-95f4a7\" (UID: \"4bec169a-6a5f-4b30-bf2a-5a83f2fce624\") " pod="opendatahub/test-curl-7134-95f4a7" Nov 01 06:33:15.793670 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:15.793642 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5p5n\" (UniqueName: \"kubernetes.io/projected/4bec169a-6a5f-4b30-bf2a-5a83f2fce624-kube-api-access-j5p5n\") pod \"test-curl-7134-95f4a7\" (UID: \"4bec169a-6a5f-4b30-bf2a-5a83f2fce624\") " pod="opendatahub/test-curl-7134-95f4a7" Nov 01 06:33:15.805574 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:15.805547 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7134-95f4a7" Nov 01 06:33:16.187635 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:16.187606 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7134-95f4a7"] Nov 01 06:33:16.188793 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:33:16.188768 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4bec169a_6a5f_4b30_bf2a_5a83f2fce624.slice/crio-e93325701dc5fc97b8cd6594431f30e4bfd899a0e7f9efd7e75a7fcc8448dc15 WatchSource:0}: Error finding container e93325701dc5fc97b8cd6594431f30e4bfd899a0e7f9efd7e75a7fcc8448dc15: Status 404 returned error can't find the container with id e93325701dc5fc97b8cd6594431f30e4bfd899a0e7f9efd7e75a7fcc8448dc15 Nov 01 06:33:17.183206 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:17.183172 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7134-95f4a7" event={"ID":"4bec169a-6a5f-4b30-bf2a-5a83f2fce624","Type":"ContainerStarted","Data":"acb0872ac966c86c9ec00cf1f9085c59b2e67c3b3193b2fb53c9965841560cb2"} Nov 01 06:33:17.183618 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:17.183215 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7134-95f4a7" event={"ID":"4bec169a-6a5f-4b30-bf2a-5a83f2fce624","Type":"ContainerStarted","Data":"e93325701dc5fc97b8cd6594431f30e4bfd899a0e7f9efd7e75a7fcc8448dc15"} Nov 01 06:33:17.299308 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:17.299255 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7134-95f4a7" podStartSLOduration=2.299241758 podStartE2EDuration="2.299241758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:33:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:33:17.297730167 +0000 UTC m=+2875.952847333" watchObservedRunningTime="2026-11-01 06:33:17.299241758 +0000 UTC m=+2875.954358925" Nov 01 06:33:18.810502 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:18.805542 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7134-95f4a7"] Nov 01 06:33:18.811115 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:18.811088 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7134-95f4a7"] Nov 01 06:33:19.195833 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:19.195740 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7134-95f4a7" podUID="4bec169a-6a5f-4b30-bf2a-5a83f2fce624" containerName="test-curl-7134-95f4a7" containerID="cri-o://acb0872ac966c86c9ec00cf1f9085c59b2e67c3b3193b2fb53c9965841560cb2" gracePeriod=2 Nov 01 06:33:19.247657 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:33:19.247596 5040 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7134-95f4a7\" is forbidden: User \"system:node:ip-10-0-143-195.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-195.us-west-1.compute.internal' and this object" podUID="4bec169a-6a5f-4b30-bf2a-5a83f2fce624" pod="opendatahub/test-curl-7134-95f4a7" Nov 01 06:33:19.944146 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:19.944108 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-qtnvw"] Nov 01 06:33:19.944676 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:19.944659 5040 cpu_manager.go:414] "containerMap: removing container" podUID="4bec169a-6a5f-4b30-bf2a-5a83f2fce624" containerName="test-curl-7134-95f4a7" Nov 01 06:33:19.944726 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:19.944677 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bec169a-6a5f-4b30-bf2a-5a83f2fce624" containerName="test-curl-7134-95f4a7" Nov 01 06:33:19.944778 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:19.944767 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bec169a-6a5f-4b30-bf2a-5a83f2fce624" containerName="test-curl-7134-95f4a7" Nov 01 06:33:19.947926 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:19.947910 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-qtnvw" Nov 01 06:33:19.999407 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:19.999372 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cdf9878-866f-49b0-b6a6-d59596787b11-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c0662f25-qtnvw\" (UID: \"7cdf9878-866f-49b0-b6a6-d59596787b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-qtnvw" Nov 01 06:33:19.999622 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:19.999444 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cdf9878-866f-49b0-b6a6-d59596787b11-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c0662f25-qtnvw\" (UID: \"7cdf9878-866f-49b0-b6a6-d59596787b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-qtnvw" Nov 01 06:33:19.999622 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:19.999606 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twzr9\" (UniqueName: \"kubernetes.io/projected/7cdf9878-866f-49b0-b6a6-d59596787b11-kube-api-access-twzr9\") pod \"maas-api-revoke-keys-e2e-mt-c0662f25-qtnvw\" (UID: \"7cdf9878-866f-49b0-b6a6-d59596787b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-qtnvw" Nov 01 06:33:20.052092 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:20.052053 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:33:20.052092 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:20.052079 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:33:20.052092 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:20.052102 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:33:20.052377 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:20.052245 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:33:20.055239 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:20.055210 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-qtnvw"] Nov 01 06:33:20.100757 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:20.100725 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twzr9\" (UniqueName: \"kubernetes.io/projected/7cdf9878-866f-49b0-b6a6-d59596787b11-kube-api-access-twzr9\") pod \"maas-api-revoke-keys-e2e-mt-c0662f25-qtnvw\" (UID: \"7cdf9878-866f-49b0-b6a6-d59596787b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-qtnvw" Nov 01 06:33:20.100951 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:20.100824 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cdf9878-866f-49b0-b6a6-d59596787b11-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c0662f25-qtnvw\" (UID: \"7cdf9878-866f-49b0-b6a6-d59596787b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-qtnvw" Nov 01 06:33:20.100951 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:20.100874 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cdf9878-866f-49b0-b6a6-d59596787b11-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c0662f25-qtnvw\" (UID: \"7cdf9878-866f-49b0-b6a6-d59596787b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-qtnvw" Nov 01 06:33:20.101498 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:20.101464 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cdf9878-866f-49b0-b6a6-d59596787b11-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c0662f25-qtnvw\" (UID: \"7cdf9878-866f-49b0-b6a6-d59596787b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-qtnvw" Nov 01 06:33:20.103194 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:20.103170 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cdf9878-866f-49b0-b6a6-d59596787b11-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c0662f25-qtnvw\" (UID: \"7cdf9878-866f-49b0-b6a6-d59596787b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-qtnvw" Nov 01 06:33:20.165265 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:20.165228 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twzr9\" (UniqueName: \"kubernetes.io/projected/7cdf9878-866f-49b0-b6a6-d59596787b11-kube-api-access-twzr9\") pod \"maas-api-revoke-keys-e2e-mt-c0662f25-qtnvw\" (UID: \"7cdf9878-866f-49b0-b6a6-d59596787b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-qtnvw" Nov 01 06:33:20.261081 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:20.261053 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-qtnvw" Nov 01 06:33:20.636161 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:20.636133 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-qtnvw"] Nov 01 06:33:20.637309 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:33:20.637284 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cdf9878_866f_49b0_b6a6_d59596787b11.slice/crio-5086f6f6aac76f7eeed2c96a271b1cc4fcce8bea315dbcbc1061848adc1e817c WatchSource:0}: Error finding container 5086f6f6aac76f7eeed2c96a271b1cc4fcce8bea315dbcbc1061848adc1e817c: Status 404 returned error can't find the container with id 5086f6f6aac76f7eeed2c96a271b1cc4fcce8bea315dbcbc1061848adc1e817c Nov 01 06:33:21.211085 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:21.211057 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-qtnvw" event={"ID":"7cdf9878-866f-49b0-b6a6-d59596787b11","Type":"ContainerStarted","Data":"ddc9231b454a02f9676e3a88bd37775e562929fc2f60bcd0b4ad1c8b81810f95"} Nov 01 06:33:21.211385 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:21.211110 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-qtnvw" event={"ID":"7cdf9878-866f-49b0-b6a6-d59596787b11","Type":"ContainerStarted","Data":"5086f6f6aac76f7eeed2c96a271b1cc4fcce8bea315dbcbc1061848adc1e817c"} Nov 01 06:33:21.328067 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:21.328011 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-qtnvw" podStartSLOduration=2.327999074 podStartE2EDuration="2.327999074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:33:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:33:21.327391887 +0000 UTC m=+2879.982509052" watchObservedRunningTime="2026-11-01 06:33:21.327999074 +0000 UTC m=+2879.983116286" Nov 01 06:33:21.451803 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:21.451780 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7134-95f4a7" Nov 01 06:33:21.516175 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:21.516150 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bec169a-6a5f-4b30-bf2a-5a83f2fce624-kube-api-access-j5p5n\" (UniqueName: \"kubernetes.io/projected/4bec169a-6a5f-4b30-bf2a-5a83f2fce624-kube-api-access-j5p5n\") pod \"4bec169a-6a5f-4b30-bf2a-5a83f2fce624\" (UID: \"4bec169a-6a5f-4b30-bf2a-5a83f2fce624\") " Nov 01 06:33:21.517955 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:21.517935 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bec169a-6a5f-4b30-bf2a-5a83f2fce624-kube-api-access-j5p5n" pod "4bec169a-6a5f-4b30-bf2a-5a83f2fce624" (UID: "4bec169a-6a5f-4b30-bf2a-5a83f2fce624"). InnerVolumeSpecName "kube-api-access-j5p5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:33:21.617796 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:21.617769 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5p5n\" (UniqueName: \"kubernetes.io/projected/4bec169a-6a5f-4b30-bf2a-5a83f2fce624-kube-api-access-j5p5n\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:22.219088 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:22.219051 5040 generic.go:356] "Generic (PLEG): container finished" podID="7cdf9878-866f-49b0-b6a6-d59596787b11" containerID="ddc9231b454a02f9676e3a88bd37775e562929fc2f60bcd0b4ad1c8b81810f95" exitCode=0 Nov 01 06:33:22.219518 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:22.219146 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-qtnvw" event={"ID":"7cdf9878-866f-49b0-b6a6-d59596787b11","Type":"ContainerDied","Data":"ddc9231b454a02f9676e3a88bd37775e562929fc2f60bcd0b4ad1c8b81810f95"} Nov 01 06:33:22.220379 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:22.220355 5040 generic.go:356] "Generic (PLEG): container finished" podID="4bec169a-6a5f-4b30-bf2a-5a83f2fce624" containerID="acb0872ac966c86c9ec00cf1f9085c59b2e67c3b3193b2fb53c9965841560cb2" exitCode=137 Nov 01 06:33:22.220428 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:22.220410 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7134-95f4a7" Nov 01 06:33:22.220467 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:22.220451 5040 scope.go:122] "RemoveContainer" containerID="acb0872ac966c86c9ec00cf1f9085c59b2e67c3b3193b2fb53c9965841560cb2" Nov 01 06:33:22.229953 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:22.229936 5040 scope.go:122] "RemoveContainer" containerID="acb0872ac966c86c9ec00cf1f9085c59b2e67c3b3193b2fb53c9965841560cb2" Nov 01 06:33:22.230202 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:33:22.230184 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acb0872ac966c86c9ec00cf1f9085c59b2e67c3b3193b2fb53c9965841560cb2\": container with ID starting with acb0872ac966c86c9ec00cf1f9085c59b2e67c3b3193b2fb53c9965841560cb2 not found: ID does not exist" containerID="acb0872ac966c86c9ec00cf1f9085c59b2e67c3b3193b2fb53c9965841560cb2" Nov 01 06:33:22.230259 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:22.230209 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acb0872ac966c86c9ec00cf1f9085c59b2e67c3b3193b2fb53c9965841560cb2"} err="failed to get container status \"acb0872ac966c86c9ec00cf1f9085c59b2e67c3b3193b2fb53c9965841560cb2\": rpc error: code = NotFound desc = could not find container \"acb0872ac966c86c9ec00cf1f9085c59b2e67c3b3193b2fb53c9965841560cb2\": container with ID starting with acb0872ac966c86c9ec00cf1f9085c59b2e67c3b3193b2fb53c9965841560cb2 not found: ID does not exist" Nov 01 06:33:23.277840 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:23.277802 5040 scope.go:122] "RemoveContainer" containerID="9205ca411e791d84fad2b3bbf56a46cbab6e5a7fbbbc81d699ab141870b5ed6b" Nov 01 06:33:23.395334 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:23.395306 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-qtnvw" Nov 01 06:33:23.437457 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:23.437419 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cdf9878-866f-49b0-b6a6-d59596787b11-kube-api-access-twzr9\" (UniqueName: \"kubernetes.io/projected/7cdf9878-866f-49b0-b6a6-d59596787b11-kube-api-access-twzr9\") pod \"7cdf9878-866f-49b0-b6a6-d59596787b11\" (UID: \"7cdf9878-866f-49b0-b6a6-d59596787b11\") " Nov 01 06:33:23.437660 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:23.437461 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cdf9878-866f-49b0-b6a6-d59596787b11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cdf9878-866f-49b0-b6a6-d59596787b11-maas-api-openshift-ca\") pod \"7cdf9878-866f-49b0-b6a6-d59596787b11\" (UID: \"7cdf9878-866f-49b0-b6a6-d59596787b11\") " Nov 01 06:33:23.437660 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:23.437573 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cdf9878-866f-49b0-b6a6-d59596787b11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cdf9878-866f-49b0-b6a6-d59596787b11-maas-api-xks-ca\") pod \"7cdf9878-866f-49b0-b6a6-d59596787b11\" (UID: \"7cdf9878-866f-49b0-b6a6-d59596787b11\") " Nov 01 06:33:23.437863 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:23.437838 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cdf9878-866f-49b0-b6a6-d59596787b11-maas-api-openshift-ca" pod "7cdf9878-866f-49b0-b6a6-d59596787b11" (UID: "7cdf9878-866f-49b0-b6a6-d59596787b11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:33:23.438055 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:23.438036 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cdf9878-866f-49b0-b6a6-d59596787b11-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:23.439497 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:23.439452 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cdf9878-866f-49b0-b6a6-d59596787b11-maas-api-xks-ca" pod "7cdf9878-866f-49b0-b6a6-d59596787b11" (UID: "7cdf9878-866f-49b0-b6a6-d59596787b11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:33:23.439497 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:23.439459 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cdf9878-866f-49b0-b6a6-d59596787b11-kube-api-access-twzr9" pod "7cdf9878-866f-49b0-b6a6-d59596787b11" (UID: "7cdf9878-866f-49b0-b6a6-d59596787b11"). InnerVolumeSpecName "kube-api-access-twzr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:33:23.516251 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:23.516214 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bec169a-6a5f-4b30-bf2a-5a83f2fce624" path="/var/lib/kubelet/pods/4bec169a-6a5f-4b30-bf2a-5a83f2fce624/volumes" Nov 01 06:33:23.538951 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:23.538923 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twzr9\" (UniqueName: \"kubernetes.io/projected/7cdf9878-866f-49b0-b6a6-d59596787b11-kube-api-access-twzr9\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:23.538951 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:23.538951 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cdf9878-866f-49b0-b6a6-d59596787b11-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:24.236424 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:24.236397 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-qtnvw" Nov 01 06:33:24.236635 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:24.236397 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-qtnvw" event={"ID":"7cdf9878-866f-49b0-b6a6-d59596787b11","Type":"ContainerDied","Data":"5086f6f6aac76f7eeed2c96a271b1cc4fcce8bea315dbcbc1061848adc1e817c"} Nov 01 06:33:24.236635 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:24.236528 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5086f6f6aac76f7eeed2c96a271b1cc4fcce8bea315dbcbc1061848adc1e817c" Nov 01 06:33:35.413949 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:35.413907 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7125-72ac69"] Nov 01 06:33:35.414667 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:35.414645 5040 cpu_manager.go:414] "containerMap: removing container" podUID="7cdf9878-866f-49b0-b6a6-d59596787b11" containerName="revoke-keys" Nov 01 06:33:35.414667 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:35.414668 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cdf9878-866f-49b0-b6a6-d59596787b11" containerName="revoke-keys" Nov 01 06:33:35.414940 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:35.414848 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cdf9878-866f-49b0-b6a6-d59596787b11" containerName="revoke-keys" Nov 01 06:33:35.418412 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:35.418389 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7125-72ac69" Nov 01 06:33:35.473325 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:35.473291 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7125-72ac69"] Nov 01 06:33:35.483054 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:35.483026 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk4qc\" (UniqueName: \"kubernetes.io/projected/0270a526-04af-4f7e-8836-105f8f61f9c7-kube-api-access-nk4qc\") pod \"test-curl-7125-72ac69\" (UID: \"0270a526-04af-4f7e-8836-105f8f61f9c7\") " pod="opendatahub/test-curl-7125-72ac69" Nov 01 06:33:35.488344 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:35.488322 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hgn5l\"" Nov 01 06:33:35.584540 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:35.584503 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk4qc\" (UniqueName: \"kubernetes.io/projected/0270a526-04af-4f7e-8836-105f8f61f9c7-kube-api-access-nk4qc\") pod \"test-curl-7125-72ac69\" (UID: \"0270a526-04af-4f7e-8836-105f8f61f9c7\") " pod="opendatahub/test-curl-7125-72ac69" Nov 01 06:33:35.646079 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:35.646048 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk4qc\" (UniqueName: \"kubernetes.io/projected/0270a526-04af-4f7e-8836-105f8f61f9c7-kube-api-access-nk4qc\") pod \"test-curl-7125-72ac69\" (UID: \"0270a526-04af-4f7e-8836-105f8f61f9c7\") " pod="opendatahub/test-curl-7125-72ac69" Nov 01 06:33:35.731875 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:35.731790 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7125-72ac69" Nov 01 06:33:36.106201 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:36.106169 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7125-72ac69"] Nov 01 06:33:36.107426 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:33:36.107401 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0270a526_04af_4f7e_8836_105f8f61f9c7.slice/crio-4d12342e17d0912a5d6ba12fe729234161d3c1dfd99c971383b9b20809e8bf35 WatchSource:0}: Error finding container 4d12342e17d0912a5d6ba12fe729234161d3c1dfd99c971383b9b20809e8bf35: Status 404 returned error can't find the container with id 4d12342e17d0912a5d6ba12fe729234161d3c1dfd99c971383b9b20809e8bf35 Nov 01 06:33:36.328042 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:36.328008 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7125-72ac69" event={"ID":"0270a526-04af-4f7e-8836-105f8f61f9c7","Type":"ContainerStarted","Data":"1d2b99add31ce3ef2114c38050493858a0e42519e983e73a60ab796ba5148181"} Nov 01 06:33:36.328216 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:36.328050 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7125-72ac69" event={"ID":"0270a526-04af-4f7e-8836-105f8f61f9c7","Type":"ContainerStarted","Data":"4d12342e17d0912a5d6ba12fe729234161d3c1dfd99c971383b9b20809e8bf35"} Nov 01 06:33:36.444754 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:36.444655 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7125-72ac69" podStartSLOduration=1.444643474 podStartE2EDuration="1.444643474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:33:36.442335344 +0000 UTC m=+2895.097452511" watchObservedRunningTime="2026-11-01 06:33:36.444643474 +0000 UTC m=+2895.099760640" Nov 01 06:33:37.845367 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:37.845330 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7125-72ac69"] Nov 01 06:33:37.849824 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:37.849786 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7125-72ac69"] Nov 01 06:33:37.987427 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:37.987388 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7125-62a89f"] Nov 01 06:33:37.987899 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:37.987884 5040 cpu_manager.go:414] "containerMap: removing container" podUID="0270a526-04af-4f7e-8836-105f8f61f9c7" containerName="test-curl-7125-72ac69" Nov 01 06:33:37.987947 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:37.987901 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0270a526-04af-4f7e-8836-105f8f61f9c7" containerName="test-curl-7125-72ac69" Nov 01 06:33:37.988035 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:37.988026 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="0270a526-04af-4f7e-8836-105f8f61f9c7" containerName="test-curl-7125-72ac69" Nov 01 06:33:37.991410 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:37.991394 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7125-62a89f" Nov 01 06:33:38.047916 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:38.047883 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7125-62a89f"] Nov 01 06:33:38.109614 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:38.109531 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv6s5\" (UniqueName: \"kubernetes.io/projected/8552e4ac-89a8-4b27-ac55-09459f043d06-kube-api-access-qv6s5\") pod \"test-curl-7125-62a89f\" (UID: \"8552e4ac-89a8-4b27-ac55-09459f043d06\") " pod="opendatahub/test-curl-7125-62a89f" Nov 01 06:33:38.210993 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:38.210953 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv6s5\" (UniqueName: \"kubernetes.io/projected/8552e4ac-89a8-4b27-ac55-09459f043d06-kube-api-access-qv6s5\") pod \"test-curl-7125-62a89f\" (UID: \"8552e4ac-89a8-4b27-ac55-09459f043d06\") " pod="opendatahub/test-curl-7125-62a89f" Nov 01 06:33:38.272337 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:38.272302 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv6s5\" (UniqueName: \"kubernetes.io/projected/8552e4ac-89a8-4b27-ac55-09459f043d06-kube-api-access-qv6s5\") pod \"test-curl-7125-62a89f\" (UID: \"8552e4ac-89a8-4b27-ac55-09459f043d06\") " pod="opendatahub/test-curl-7125-62a89f" Nov 01 06:33:38.303854 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:38.303823 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7125-62a89f" Nov 01 06:33:38.342347 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:38.342275 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7125-72ac69" podUID="0270a526-04af-4f7e-8836-105f8f61f9c7" containerName="test-curl-7125-72ac69" containerID="cri-o://1d2b99add31ce3ef2114c38050493858a0e42519e983e73a60ab796ba5148181" gracePeriod=2 Nov 01 06:33:38.394075 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:33:38.394036 5040 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7125-72ac69\" is forbidden: User \"system:node:ip-10-0-143-195.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-195.us-west-1.compute.internal' and this object" podUID="0270a526-04af-4f7e-8836-105f8f61f9c7" pod="opendatahub/test-curl-7125-72ac69" Nov 01 06:33:38.677045 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:38.677010 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7125-62a89f"] Nov 01 06:33:38.677645 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:33:38.677619 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8552e4ac_89a8_4b27_ac55_09459f043d06.slice/crio-2f576aca35b90d013c93dabf06d7e56d91264b94da23c3925990ba7ed4f2b61c WatchSource:0}: Error finding container 2f576aca35b90d013c93dabf06d7e56d91264b94da23c3925990ba7ed4f2b61c: Status 404 returned error can't find the container with id 2f576aca35b90d013c93dabf06d7e56d91264b94da23c3925990ba7ed4f2b61c Nov 01 06:33:39.355692 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:39.355649 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7125-62a89f" event={"ID":"8552e4ac-89a8-4b27-ac55-09459f043d06","Type":"ContainerStarted","Data":"3f14c5e1e63119ea985a6bf140090a411931111dfb5ecf9d5254ddf28cdc6aa3"} Nov 01 06:33:39.356132 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:39.355705 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7125-62a89f" event={"ID":"8552e4ac-89a8-4b27-ac55-09459f043d06","Type":"ContainerStarted","Data":"2f576aca35b90d013c93dabf06d7e56d91264b94da23c3925990ba7ed4f2b61c"} Nov 01 06:33:39.472305 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:39.472252 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7125-62a89f" podStartSLOduration=2.472231931 podStartE2EDuration="2.472231931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:33:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:33:39.4700331 +0000 UTC m=+2898.125150270" watchObservedRunningTime="2026-11-01 06:33:39.472231931 +0000 UTC m=+2898.127349097" Nov 01 06:33:39.521768 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:33:39.521726 5040 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7125-72ac69\" is forbidden: User \"system:node:ip-10-0-143-195.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-195.us-west-1.compute.internal' and this object" podUID="0270a526-04af-4f7e-8836-105f8f61f9c7" pod="opendatahub/test-curl-7125-72ac69" Nov 01 06:33:40.024507 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:33:40.020986 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:33:40.614583 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:40.614515 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7125-72ac69" Nov 01 06:33:40.744067 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:40.744028 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0270a526-04af-4f7e-8836-105f8f61f9c7-kube-api-access-nk4qc\" (UniqueName: \"kubernetes.io/projected/0270a526-04af-4f7e-8836-105f8f61f9c7-kube-api-access-nk4qc\") pod \"0270a526-04af-4f7e-8836-105f8f61f9c7\" (UID: \"0270a526-04af-4f7e-8836-105f8f61f9c7\") " Nov 01 06:33:40.746095 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:40.746066 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0270a526-04af-4f7e-8836-105f8f61f9c7-kube-api-access-nk4qc" pod "0270a526-04af-4f7e-8836-105f8f61f9c7" (UID: "0270a526-04af-4f7e-8836-105f8f61f9c7"). InnerVolumeSpecName "kube-api-access-nk4qc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:33:40.846563 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:40.846468 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk4qc\" (UniqueName: \"kubernetes.io/projected/0270a526-04af-4f7e-8836-105f8f61f9c7-kube-api-access-nk4qc\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:40.977683 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:40.977441 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7125-62a89f"] Nov 01 06:33:40.982749 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:40.982683 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7125-62a89f"] Nov 01 06:33:41.151506 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:41.147761 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7125-d322ca"] Nov 01 06:33:41.151506 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:41.148297 5040 cpu_manager.go:414] "containerMap: removing container" podUID="8552e4ac-89a8-4b27-ac55-09459f043d06" containerName="test-curl-7125-62a89f" Nov 01 06:33:41.151506 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:41.148311 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8552e4ac-89a8-4b27-ac55-09459f043d06" containerName="test-curl-7125-62a89f" Nov 01 06:33:41.151506 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:41.148433 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="8552e4ac-89a8-4b27-ac55-09459f043d06" containerName="test-curl-7125-62a89f" Nov 01 06:33:41.153091 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:41.153048 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7125-d322ca" Nov 01 06:33:41.178960 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:41.178930 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zww97\" (UniqueName: \"kubernetes.io/projected/d9a76907-1aec-4ea1-b6c8-4ac442a9007a-kube-api-access-zww97\") pod \"test-curl-7125-d322ca\" (UID: \"d9a76907-1aec-4ea1-b6c8-4ac442a9007a\") " pod="opendatahub/test-curl-7125-d322ca" Nov 01 06:33:41.189861 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:41.189831 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7125-d322ca"] Nov 01 06:33:41.284694 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:41.280266 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zww97\" (UniqueName: \"kubernetes.io/projected/d9a76907-1aec-4ea1-b6c8-4ac442a9007a-kube-api-access-zww97\") pod \"test-curl-7125-d322ca\" (UID: \"d9a76907-1aec-4ea1-b6c8-4ac442a9007a\") " pod="opendatahub/test-curl-7125-d322ca" Nov 01 06:33:41.344938 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:41.344897 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zww97\" (UniqueName: \"kubernetes.io/projected/d9a76907-1aec-4ea1-b6c8-4ac442a9007a-kube-api-access-zww97\") pod \"test-curl-7125-d322ca\" (UID: \"d9a76907-1aec-4ea1-b6c8-4ac442a9007a\") " pod="opendatahub/test-curl-7125-d322ca" Nov 01 06:33:41.372176 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:41.372145 5040 generic.go:356] "Generic (PLEG): container finished" podID="0270a526-04af-4f7e-8836-105f8f61f9c7" containerID="1d2b99add31ce3ef2114c38050493858a0e42519e983e73a60ab796ba5148181" exitCode=137 Nov 01 06:33:41.372420 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:41.372396 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7125-72ac69" Nov 01 06:33:41.372649 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:41.372629 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7125-62a89f" podUID="8552e4ac-89a8-4b27-ac55-09459f043d06" containerName="test-curl-7125-62a89f" containerID="cri-o://3f14c5e1e63119ea985a6bf140090a411931111dfb5ecf9d5254ddf28cdc6aa3" gracePeriod=2 Nov 01 06:33:41.372850 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:41.372311 5040 scope.go:122] "RemoveContainer" containerID="1d2b99add31ce3ef2114c38050493858a0e42519e983e73a60ab796ba5148181" Nov 01 06:33:41.387626 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:41.387598 5040 scope.go:122] "RemoveContainer" containerID="1d2b99add31ce3ef2114c38050493858a0e42519e983e73a60ab796ba5148181" Nov 01 06:33:41.388013 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:33:41.387990 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d2b99add31ce3ef2114c38050493858a0e42519e983e73a60ab796ba5148181\": container with ID starting with 1d2b99add31ce3ef2114c38050493858a0e42519e983e73a60ab796ba5148181 not found: ID does not exist" containerID="1d2b99add31ce3ef2114c38050493858a0e42519e983e73a60ab796ba5148181" Nov 01 06:33:41.388122 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:41.388020 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d2b99add31ce3ef2114c38050493858a0e42519e983e73a60ab796ba5148181"} err="failed to get container status \"1d2b99add31ce3ef2114c38050493858a0e42519e983e73a60ab796ba5148181\": rpc error: code = NotFound desc = could not find container \"1d2b99add31ce3ef2114c38050493858a0e42519e983e73a60ab796ba5148181\": container with ID starting with 1d2b99add31ce3ef2114c38050493858a0e42519e983e73a60ab796ba5148181 not found: ID does not exist" Nov 01 06:33:41.428507 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:33:41.426704 5040 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7125-62a89f\" is forbidden: User \"system:node:ip-10-0-143-195.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-195.us-west-1.compute.internal' and this object" podUID="8552e4ac-89a8-4b27-ac55-09459f043d06" pod="opendatahub/test-curl-7125-62a89f" Nov 01 06:33:41.471886 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:41.471852 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7125-d322ca" Nov 01 06:33:41.480660 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:33:41.480623 5040 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7125-72ac69\" is forbidden: User \"system:node:ip-10-0-143-195.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-195.us-west-1.compute.internal' and this object" podUID="0270a526-04af-4f7e-8836-105f8f61f9c7" pod="opendatahub/test-curl-7125-72ac69" Nov 01 06:33:41.539595 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:33:41.536747 5040 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7125-62a89f\" is forbidden: User \"system:node:ip-10-0-143-195.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-195.us-west-1.compute.internal' and this object" podUID="8552e4ac-89a8-4b27-ac55-09459f043d06" pod="opendatahub/test-curl-7125-62a89f" Nov 01 06:33:41.580511 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:41.579999 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0270a526-04af-4f7e-8836-105f8f61f9c7" path="/var/lib/kubelet/pods/0270a526-04af-4f7e-8836-105f8f61f9c7/volumes" Nov 01 06:33:41.636909 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:33:41.636737 5040 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7125-62a89f\" is forbidden: User \"system:node:ip-10-0-143-195.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-195.us-west-1.compute.internal' and this object" podUID="8552e4ac-89a8-4b27-ac55-09459f043d06" pod="opendatahub/test-curl-7125-62a89f" Nov 01 06:33:41.692325 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:33:41.691173 5040 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7125-72ac69\" is forbidden: User \"system:node:ip-10-0-143-195.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-195.us-west-1.compute.internal' and this object" podUID="0270a526-04af-4f7e-8836-105f8f61f9c7" pod="opendatahub/test-curl-7125-72ac69" Nov 01 06:33:42.061930 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:42.050680 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7125-d322ca"] Nov 01 06:33:42.061930 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:33:42.051704 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9a76907_1aec_4ea1_b6c8_4ac442a9007a.slice/crio-ce04e6723a04b7fb38a8d004fb032a3961f9c2dffd912318b73311c197285a9e WatchSource:0}: Error finding container ce04e6723a04b7fb38a8d004fb032a3961f9c2dffd912318b73311c197285a9e: Status 404 returned error can't find the container with id ce04e6723a04b7fb38a8d004fb032a3961f9c2dffd912318b73311c197285a9e Nov 01 06:33:42.393526 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:42.392416 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7125-d322ca" event={"ID":"d9a76907-1aec-4ea1-b6c8-4ac442a9007a","Type":"ContainerStarted","Data":"15229441da946409381bcdadc8fa0425194123b926b89bc2e77125df9321179e"} Nov 01 06:33:42.393526 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:42.392495 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7125-d322ca" event={"ID":"d9a76907-1aec-4ea1-b6c8-4ac442a9007a","Type":"ContainerStarted","Data":"ce04e6723a04b7fb38a8d004fb032a3961f9c2dffd912318b73311c197285a9e"} Nov 01 06:33:43.380635 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:43.380589 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" podUID="75afe340-0b9c-4b4c-89ab-20d761e6b4f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:33:43.381040 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:43.380987 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" podUID="21f54836-d1ed-4f87-8f26-7c2d939b2590" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Nov 01 06:33:43.439538 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:43.425325 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" podUID="1878b6fb-f8fe-4f4f-8f4e-056c66fc206a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:33:43.496697 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:43.491011 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc" podUID="66fdf9da-4790-40c5-9434-052c7465d23e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.124:8000/ready\": context deadline exceeded" Nov 01 06:33:43.613503 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:43.609642 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" podUID="57cf5e07-36de-48cf-b938-138484cea3a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:33:44.007661 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:44.007601 5040 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" podUID="5daa03ca-824b-4799-a11b-4b3af1512c38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.123:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:33:44.278682 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:44.278570 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" podUID="5daa03ca-824b-4799-a11b-4b3af1512c38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.123:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:33:44.386504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:44.382591 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" podUID="75afe340-0b9c-4b4c-89ab-20d761e6b4f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Nov 01 06:33:44.403507 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:44.400193 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" podUID="21f54836-d1ed-4f87-8f26-7c2d939b2590" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Nov 01 06:33:44.403507 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:44.400772 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" Nov 01 06:33:44.403507 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:44.400835 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" Nov 01 06:33:44.414809 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:44.413722 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv" podUID="72ab782a-6494-430b-a5eb-17f4b4f9ddd3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": context deadline exceeded" Nov 01 06:33:44.499656 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:44.499628 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" Nov 01 06:33:44.503746 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:44.502928 5040 generic.go:356] "Generic (PLEG): container finished" podID="8552e4ac-89a8-4b27-ac55-09459f043d06" containerID="3f14c5e1e63119ea985a6bf140090a411931111dfb5ecf9d5254ddf28cdc6aa3" exitCode=137 Nov 01 06:33:44.524038 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:44.523943 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" Nov 01 06:33:44.554829 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:44.554464 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7125-d322ca" podStartSLOduration=3.5544445639999998 podStartE2EDuration="3.554444564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:33:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:33:42.526662753 +0000 UTC m=+2901.181779925" watchObservedRunningTime="2026-11-01 06:33:44.554444564 +0000 UTC m=+2903.209561730" Nov 01 06:33:44.699629 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:44.699597 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7125-62a89f" Nov 01 06:33:44.867287 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:44.866470 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8552e4ac-89a8-4b27-ac55-09459f043d06-kube-api-access-qv6s5\" (UniqueName: \"kubernetes.io/projected/8552e4ac-89a8-4b27-ac55-09459f043d06-kube-api-access-qv6s5\") pod \"8552e4ac-89a8-4b27-ac55-09459f043d06\" (UID: \"8552e4ac-89a8-4b27-ac55-09459f043d06\") " Nov 01 06:33:44.961152 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:44.960989 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8552e4ac-89a8-4b27-ac55-09459f043d06-kube-api-access-qv6s5" pod "8552e4ac-89a8-4b27-ac55-09459f043d06" (UID: "8552e4ac-89a8-4b27-ac55-09459f043d06"). InnerVolumeSpecName "kube-api-access-qv6s5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:33:44.968074 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:44.968047 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv6s5\" (UniqueName: \"kubernetes.io/projected/8552e4ac-89a8-4b27-ac55-09459f043d06-kube-api-access-qv6s5\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:45.423405 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:45.422098 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7125-d322ca"] Nov 01 06:33:45.423405 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:45.422316 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7125-d322ca" podUID="d9a76907-1aec-4ea1-b6c8-4ac442a9007a" containerName="test-curl-7125-d322ca" containerID="cri-o://15229441da946409381bcdadc8fa0425194123b926b89bc2e77125df9321179e" gracePeriod=2 Nov 01 06:33:45.426167 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:45.426140 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7125-d322ca"] Nov 01 06:33:45.474241 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:33:45.474201 5040 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7125-d322ca\" is forbidden: User \"system:node:ip-10-0-143-195.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-195.us-west-1.compute.internal' and this object" podUID="d9a76907-1aec-4ea1-b6c8-4ac442a9007a" pod="opendatahub/test-curl-7125-d322ca" Nov 01 06:33:45.526097 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:45.526051 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7125-62a89f" Nov 01 06:33:45.529757 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:45.529723 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8552e4ac-89a8-4b27-ac55-09459f043d06" path="/var/lib/kubelet/pods/8552e4ac-89a8-4b27-ac55-09459f043d06/volumes" Nov 01 06:33:45.530351 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:45.530327 5040 scope.go:122] "RemoveContainer" containerID="3f14c5e1e63119ea985a6bf140090a411931111dfb5ecf9d5254ddf28cdc6aa3" Nov 01 06:33:45.564369 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:45.564345 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7125-bf471d"] Nov 01 06:33:45.564880 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:45.564864 5040 cpu_manager.go:414] "containerMap: removing container" podUID="d9a76907-1aec-4ea1-b6c8-4ac442a9007a" containerName="test-curl-7125-d322ca" Nov 01 06:33:45.564929 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:45.564882 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9a76907-1aec-4ea1-b6c8-4ac442a9007a" containerName="test-curl-7125-d322ca" Nov 01 06:33:45.564982 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:45.564973 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9a76907-1aec-4ea1-b6c8-4ac442a9007a" containerName="test-curl-7125-d322ca" Nov 01 06:33:45.569463 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:45.569447 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7125-bf471d" Nov 01 06:33:45.578121 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:33:45.578096 5040 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7125-d322ca\" is forbidden: User \"system:node:ip-10-0-143-195.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-195.us-west-1.compute.internal' and this object" podUID="d9a76907-1aec-4ea1-b6c8-4ac442a9007a" pod="opendatahub/test-curl-7125-d322ca" Nov 01 06:33:45.624811 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:45.624774 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7125-bf471d"] Nov 01 06:33:45.675010 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:45.674928 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxs68\" (UniqueName: \"kubernetes.io/projected/10a8a251-1ca1-499d-aac6-b1123f9c19ac-kube-api-access-rxs68\") pod \"test-curl-7125-bf471d\" (UID: \"10a8a251-1ca1-499d-aac6-b1123f9c19ac\") " pod="opendatahub/test-curl-7125-bf471d" Nov 01 06:33:45.775955 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:45.775918 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxs68\" (UniqueName: \"kubernetes.io/projected/10a8a251-1ca1-499d-aac6-b1123f9c19ac-kube-api-access-rxs68\") pod \"test-curl-7125-bf471d\" (UID: \"10a8a251-1ca1-499d-aac6-b1123f9c19ac\") " pod="opendatahub/test-curl-7125-bf471d" Nov 01 06:33:45.837121 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:45.837082 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxs68\" (UniqueName: \"kubernetes.io/projected/10a8a251-1ca1-499d-aac6-b1123f9c19ac-kube-api-access-rxs68\") pod \"test-curl-7125-bf471d\" (UID: \"10a8a251-1ca1-499d-aac6-b1123f9c19ac\") " pod="opendatahub/test-curl-7125-bf471d" Nov 01 06:33:45.881343 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:45.881308 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7125-bf471d" Nov 01 06:33:46.320516 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:46.320458 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7125-bf471d"] Nov 01 06:33:46.321927 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:33:46.321884 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10a8a251_1ca1_499d_aac6_b1123f9c19ac.slice/crio-ab4cec8d41dee94514851e8f41d0569bedf38b932d2c6ae865beea9ed9179146 WatchSource:0}: Error finding container ab4cec8d41dee94514851e8f41d0569bedf38b932d2c6ae865beea9ed9179146: Status 404 returned error can't find the container with id ab4cec8d41dee94514851e8f41d0569bedf38b932d2c6ae865beea9ed9179146 Nov 01 06:33:46.537193 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:46.537147 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7125-bf471d" event={"ID":"10a8a251-1ca1-499d-aac6-b1123f9c19ac","Type":"ContainerStarted","Data":"f2d67329a3fc9c52a3907ef210d19b14b12ac6c0f6f5ba3f9843568988322d25"} Nov 01 06:33:46.537611 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:46.537198 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7125-bf471d" event={"ID":"10a8a251-1ca1-499d-aac6-b1123f9c19ac","Type":"ContainerStarted","Data":"ab4cec8d41dee94514851e8f41d0569bedf38b932d2c6ae865beea9ed9179146"} Nov 01 06:33:46.652686 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:46.652561 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7125-bf471d" podStartSLOduration=1.652546934 podStartE2EDuration="1.652546934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:33:46.651760278 +0000 UTC m=+2905.306877446" watchObservedRunningTime="2026-11-01 06:33:46.652546934 +0000 UTC m=+2905.307664127" Nov 01 06:33:47.546456 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:47.546424 5040 generic.go:356] "Generic (PLEG): container finished" podID="d9a76907-1aec-4ea1-b6c8-4ac442a9007a" containerID="15229441da946409381bcdadc8fa0425194123b926b89bc2e77125df9321179e" exitCode=137 Nov 01 06:33:47.686808 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:47.686782 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7125-d322ca" Nov 01 06:33:47.837412 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:47.837377 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9a76907-1aec-4ea1-b6c8-4ac442a9007a-kube-api-access-zww97\" (UniqueName: \"kubernetes.io/projected/d9a76907-1aec-4ea1-b6c8-4ac442a9007a-kube-api-access-zww97\") pod \"d9a76907-1aec-4ea1-b6c8-4ac442a9007a\" (UID: \"d9a76907-1aec-4ea1-b6c8-4ac442a9007a\") " Nov 01 06:33:47.839464 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:47.839432 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9a76907-1aec-4ea1-b6c8-4ac442a9007a-kube-api-access-zww97" pod "d9a76907-1aec-4ea1-b6c8-4ac442a9007a" (UID: "d9a76907-1aec-4ea1-b6c8-4ac442a9007a"). InnerVolumeSpecName "kube-api-access-zww97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:33:47.939275 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:47.939232 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zww97\" (UniqueName: \"kubernetes.io/projected/d9a76907-1aec-4ea1-b6c8-4ac442a9007a-kube-api-access-zww97\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:48.555050 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:48.555021 5040 scope.go:122] "RemoveContainer" containerID="15229441da946409381bcdadc8fa0425194123b926b89bc2e77125df9321179e" Nov 01 06:33:48.555050 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:48.555039 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7125-d322ca" Nov 01 06:33:48.618986 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:33:48.618946 5040 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7125-d322ca\" is forbidden: User \"system:node:ip-10-0-143-195.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-195.us-west-1.compute.internal' and this object" podUID="d9a76907-1aec-4ea1-b6c8-4ac442a9007a" pod="opendatahub/test-curl-7125-d322ca" Nov 01 06:33:48.780656 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:48.780608 5040 patch_prober.go:29] interesting pod/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 06:33:48.780874 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:48.780680 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" podUID="d9577376-97a1-4915-b69e-26e60c29fce4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 06:33:49.523125 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:49.523080 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9a76907-1aec-4ea1-b6c8-4ac442a9007a" path="/var/lib/kubelet/pods/d9a76907-1aec-4ea1-b6c8-4ac442a9007a/volumes" Nov 01 06:33:51.577218 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:51.577185 5040 generic.go:356] "Generic (PLEG): container finished" podID="d9577376-97a1-4915-b69e-26e60c29fce4" containerID="3dfe98b97d63aa55aa6310dee9141b058482d425f4b7115638bbf0e0bd5ea214" exitCode=0 Nov 01 06:33:51.577691 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:51.577256 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" event={"ID":"d9577376-97a1-4915-b69e-26e60c29fce4","Type":"ContainerDied","Data":"3dfe98b97d63aa55aa6310dee9141b058482d425f4b7115638bbf0e0bd5ea214"} Nov 01 06:33:51.577691 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:51.577658 5040 scope.go:122] "RemoveContainer" containerID="3dfe98b97d63aa55aa6310dee9141b058482d425f4b7115638bbf0e0bd5ea214" Nov 01 06:33:51.579119 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:51.579092 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:33:51.579187 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:51.579165 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:33:51.579228 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:51.579200 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:33:52.586187 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:52.586149 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" event={"ID":"d9577376-97a1-4915-b69e-26e60c29fce4","Type":"ContainerStarted","Data":"c1beeaff29df2cac39f16cd34c4fa153761bee8150cb413aa013d910af704a19"} Nov 01 06:33:52.615451 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:52.607887 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:33:52.615451 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:52.608012 5040 patch_prober.go:29] interesting pod/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Nov 01 06:33:52.615451 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:52.608056 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" podUID="d9577376-97a1-4915-b69e-26e60c29fce4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Nov 01 06:33:53.618617 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:53.613857 5040 patch_prober.go:29] interesting pod/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Nov 01 06:33:53.618617 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:53.613936 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" podUID="d9577376-97a1-4915-b69e-26e60c29fce4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Nov 01 06:33:54.624503 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:54.610602 5040 patch_prober.go:29] interesting pod/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Nov 01 06:33:54.624503 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:54.610682 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" podUID="d9577376-97a1-4915-b69e-26e60c29fce4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Nov 01 06:33:55.597509 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:55.596819 5040 patch_prober.go:29] interesting pod/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Nov 01 06:33:55.597509 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:55.596916 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" podUID="d9577376-97a1-4915-b69e-26e60c29fce4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Nov 01 06:33:56.598503 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:56.597845 5040 patch_prober.go:29] interesting pod/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Nov 01 06:33:56.598503 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:56.597907 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" podUID="d9577376-97a1-4915-b69e-26e60c29fce4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Nov 01 06:33:57.370503 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:57.369571 5040 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv" podUID="72ab782a-6494-430b-a5eb-17f4b4f9ddd3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:33:57.412504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:57.410974 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv" podUID="72ab782a-6494-430b-a5eb-17f4b4f9ddd3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:33:57.432617 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:57.432564 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" podUID="1878b6fb-f8fe-4f4f-8f4e-056c66fc206a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:33:57.477979 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:57.477928 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc" podUID="66fdf9da-4790-40c5-9434-052c7465d23e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.124:8000/ready\": context deadline exceeded" Nov 01 06:33:57.505620 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:57.505568 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" podUID="adbac784-4ded-4ca3-b985-3e59983fc637" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:33:57.518923 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:57.518878 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" podUID="21f54836-d1ed-4f87-8f26-7c2d939b2590" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:33:57.587503 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:57.586568 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" podUID="57cf5e07-36de-48cf-b938-138484cea3a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:33:57.607514 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:57.606881 5040 patch_prober.go:29] interesting pod/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Nov 01 06:33:57.607514 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:57.606946 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" podUID="d9577376-97a1-4915-b69e-26e60c29fce4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Nov 01 06:33:58.275683 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:58.275620 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" podUID="5daa03ca-824b-4799-a11b-4b3af1512c38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.123:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:33:58.647540 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:58.647417 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:33:58.648408 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:58.648380 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:33:58.649511 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:33:58.649447 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:34:01.816729 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:01.816625 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7125-bf471d"] Nov 01 06:34:01.817233 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:01.816915 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7125-bf471d" podUID="10a8a251-1ca1-499d-aac6-b1123f9c19ac" containerName="test-curl-7125-bf471d" containerID="cri-o://f2d67329a3fc9c52a3907ef210d19b14b12ac6c0f6f5ba3f9843568988322d25" gracePeriod=2 Nov 01 06:34:01.821978 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:01.821950 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7125-bf471d"] Nov 01 06:34:01.868979 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:34:01.868941 5040 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7125-bf471d\" is forbidden: User \"system:node:ip-10-0-143-195.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-195.us-west-1.compute.internal' and this object" podUID="10a8a251-1ca1-499d-aac6-b1123f9c19ac" pod="opendatahub/test-curl-7125-bf471d" Nov 01 06:34:03.962017 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:03.961980 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7125-857f5b"] Nov 01 06:34:03.962494 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:03.962466 5040 cpu_manager.go:414] "containerMap: removing container" podUID="10a8a251-1ca1-499d-aac6-b1123f9c19ac" containerName="test-curl-7125-bf471d" Nov 01 06:34:03.962558 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:03.962495 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10a8a251-1ca1-499d-aac6-b1123f9c19ac" containerName="test-curl-7125-bf471d" Nov 01 06:34:03.962657 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:03.962646 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="10a8a251-1ca1-499d-aac6-b1123f9c19ac" containerName="test-curl-7125-bf471d" Nov 01 06:34:03.965955 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:03.965928 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7125-857f5b" Nov 01 06:34:04.019435 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:04.019394 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7125-857f5b"] Nov 01 06:34:04.046808 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:04.046781 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7125-bf471d" Nov 01 06:34:04.098951 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:04.098882 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlff2\" (UniqueName: \"kubernetes.io/projected/9acd643c-f029-4ef6-9052-874846b71cd2-kube-api-access-dlff2\") pod \"test-curl-7125-857f5b\" (UID: \"9acd643c-f029-4ef6-9052-874846b71cd2\") " pod="opendatahub/test-curl-7125-857f5b" Nov 01 06:34:04.200116 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:04.200080 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10a8a251-1ca1-499d-aac6-b1123f9c19ac-kube-api-access-rxs68\" (UniqueName: \"kubernetes.io/projected/10a8a251-1ca1-499d-aac6-b1123f9c19ac-kube-api-access-rxs68\") pod \"10a8a251-1ca1-499d-aac6-b1123f9c19ac\" (UID: \"10a8a251-1ca1-499d-aac6-b1123f9c19ac\") " Nov 01 06:34:04.200355 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:04.200338 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlff2\" (UniqueName: \"kubernetes.io/projected/9acd643c-f029-4ef6-9052-874846b71cd2-kube-api-access-dlff2\") pod \"test-curl-7125-857f5b\" (UID: \"9acd643c-f029-4ef6-9052-874846b71cd2\") " pod="opendatahub/test-curl-7125-857f5b" Nov 01 06:34:04.202205 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:04.202178 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10a8a251-1ca1-499d-aac6-b1123f9c19ac-kube-api-access-rxs68" pod "10a8a251-1ca1-499d-aac6-b1123f9c19ac" (UID: "10a8a251-1ca1-499d-aac6-b1123f9c19ac"). InnerVolumeSpecName "kube-api-access-rxs68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:34:04.261828 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:04.261796 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlff2\" (UniqueName: \"kubernetes.io/projected/9acd643c-f029-4ef6-9052-874846b71cd2-kube-api-access-dlff2\") pod \"test-curl-7125-857f5b\" (UID: \"9acd643c-f029-4ef6-9052-874846b71cd2\") " pod="opendatahub/test-curl-7125-857f5b" Nov 01 06:34:04.277986 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:04.277957 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7125-857f5b" Nov 01 06:34:04.301102 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:04.301077 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxs68\" (UniqueName: \"kubernetes.io/projected/10a8a251-1ca1-499d-aac6-b1123f9c19ac-kube-api-access-rxs68\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:34:04.664660 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:04.660252 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7125-857f5b"] Nov 01 06:34:04.746997 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:04.746964 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7125-857f5b" event={"ID":"9acd643c-f029-4ef6-9052-874846b71cd2","Type":"ContainerStarted","Data":"34a0d07c9d6638c83190c6f9f2fdf4efc1405ea8aa95f030f729ff5a9e733f15"} Nov 01 06:34:04.748170 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:04.748147 5040 generic.go:356] "Generic (PLEG): container finished" podID="10a8a251-1ca1-499d-aac6-b1123f9c19ac" containerID="f2d67329a3fc9c52a3907ef210d19b14b12ac6c0f6f5ba3f9843568988322d25" exitCode=137 Nov 01 06:34:04.748280 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:04.748205 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7125-bf471d" Nov 01 06:34:04.748280 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:04.748245 5040 scope.go:122] "RemoveContainer" containerID="f2d67329a3fc9c52a3907ef210d19b14b12ac6c0f6f5ba3f9843568988322d25" Nov 01 06:34:04.758620 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:04.758600 5040 scope.go:122] "RemoveContainer" containerID="f2d67329a3fc9c52a3907ef210d19b14b12ac6c0f6f5ba3f9843568988322d25" Nov 01 06:34:04.758880 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:34:04.758858 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2d67329a3fc9c52a3907ef210d19b14b12ac6c0f6f5ba3f9843568988322d25\": container with ID starting with f2d67329a3fc9c52a3907ef210d19b14b12ac6c0f6f5ba3f9843568988322d25 not found: ID does not exist" containerID="f2d67329a3fc9c52a3907ef210d19b14b12ac6c0f6f5ba3f9843568988322d25" Nov 01 06:34:04.758924 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:04.758890 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2d67329a3fc9c52a3907ef210d19b14b12ac6c0f6f5ba3f9843568988322d25"} err="failed to get container status \"f2d67329a3fc9c52a3907ef210d19b14b12ac6c0f6f5ba3f9843568988322d25\": rpc error: code = NotFound desc = could not find container \"f2d67329a3fc9c52a3907ef210d19b14b12ac6c0f6f5ba3f9843568988322d25\": container with ID starting with f2d67329a3fc9c52a3907ef210d19b14b12ac6c0f6f5ba3f9843568988322d25 not found: ID does not exist" Nov 01 06:34:04.809920 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:34:04.809880 5040 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7125-bf471d\" is forbidden: User \"system:node:ip-10-0-143-195.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-195.us-west-1.compute.internal' and this object" podUID="10a8a251-1ca1-499d-aac6-b1123f9c19ac" pod="opendatahub/test-curl-7125-bf471d" Nov 01 06:34:05.520158 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:05.520128 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10a8a251-1ca1-499d-aac6-b1123f9c19ac" path="/var/lib/kubelet/pods/10a8a251-1ca1-499d-aac6-b1123f9c19ac/volumes" Nov 01 06:34:05.756921 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:05.756875 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7125-857f5b" event={"ID":"9acd643c-f029-4ef6-9052-874846b71cd2","Type":"ContainerStarted","Data":"42ad98dbe64d761d5aa69e4af0fb243d66a81aa83dca6bb579345216d5e72e17"} Nov 01 06:34:05.874321 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:05.874217 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7125-857f5b" podStartSLOduration=2.874204454 podStartE2EDuration="2.874204454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:34:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:34:05.871004687 +0000 UTC m=+2924.526121854" watchObservedRunningTime="2026-11-01 06:34:05.874204454 +0000 UTC m=+2924.529321621" Nov 01 06:34:07.377428 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:07.377395 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7125-857f5b"] Nov 01 06:34:07.380856 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:07.380830 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7125-857f5b"] Nov 01 06:34:07.771593 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:07.771523 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7125-857f5b" podUID="9acd643c-f029-4ef6-9052-874846b71cd2" containerName="test-curl-7125-857f5b" containerID="cri-o://42ad98dbe64d761d5aa69e4af0fb243d66a81aa83dca6bb579345216d5e72e17" gracePeriod=2 Nov 01 06:34:09.976390 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:34:09.976347 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:34:10.021613 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:10.021589 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7125-857f5b" Nov 01 06:34:10.171695 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:10.171610 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9acd643c-f029-4ef6-9052-874846b71cd2-kube-api-access-dlff2\" (UniqueName: \"kubernetes.io/projected/9acd643c-f029-4ef6-9052-874846b71cd2-kube-api-access-dlff2\") pod \"9acd643c-f029-4ef6-9052-874846b71cd2\" (UID: \"9acd643c-f029-4ef6-9052-874846b71cd2\") " Nov 01 06:34:10.173848 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:10.173819 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9acd643c-f029-4ef6-9052-874846b71cd2-kube-api-access-dlff2" pod "9acd643c-f029-4ef6-9052-874846b71cd2" (UID: "9acd643c-f029-4ef6-9052-874846b71cd2"). InnerVolumeSpecName "kube-api-access-dlff2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:34:10.272624 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:10.272590 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlff2\" (UniqueName: \"kubernetes.io/projected/9acd643c-f029-4ef6-9052-874846b71cd2-kube-api-access-dlff2\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:34:10.793894 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:10.793857 5040 generic.go:356] "Generic (PLEG): container finished" podID="9acd643c-f029-4ef6-9052-874846b71cd2" containerID="42ad98dbe64d761d5aa69e4af0fb243d66a81aa83dca6bb579345216d5e72e17" exitCode=137 Nov 01 06:34:10.794047 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:10.793903 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7125-857f5b" Nov 01 06:34:10.794047 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:10.793957 5040 scope.go:122] "RemoveContainer" containerID="42ad98dbe64d761d5aa69e4af0fb243d66a81aa83dca6bb579345216d5e72e17" Nov 01 06:34:10.803796 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:10.803781 5040 scope.go:122] "RemoveContainer" containerID="42ad98dbe64d761d5aa69e4af0fb243d66a81aa83dca6bb579345216d5e72e17" Nov 01 06:34:10.804042 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:34:10.804023 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42ad98dbe64d761d5aa69e4af0fb243d66a81aa83dca6bb579345216d5e72e17\": container with ID starting with 42ad98dbe64d761d5aa69e4af0fb243d66a81aa83dca6bb579345216d5e72e17 not found: ID does not exist" containerID="42ad98dbe64d761d5aa69e4af0fb243d66a81aa83dca6bb579345216d5e72e17" Nov 01 06:34:10.804114 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:10.804048 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42ad98dbe64d761d5aa69e4af0fb243d66a81aa83dca6bb579345216d5e72e17"} err="failed to get container status \"42ad98dbe64d761d5aa69e4af0fb243d66a81aa83dca6bb579345216d5e72e17\": rpc error: code = NotFound desc = could not find container \"42ad98dbe64d761d5aa69e4af0fb243d66a81aa83dca6bb579345216d5e72e17\": container with ID starting with 42ad98dbe64d761d5aa69e4af0fb243d66a81aa83dca6bb579345216d5e72e17 not found: ID does not exist" Nov 01 06:34:10.854151 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:34:10.854120 5040 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7125-857f5b\" is forbidden: User \"system:node:ip-10-0-143-195.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-195.us-west-1.compute.internal' and this object" podUID="9acd643c-f029-4ef6-9052-874846b71cd2" pod="opendatahub/test-curl-7125-857f5b" Nov 01 06:34:11.516993 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:11.516962 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9acd643c-f029-4ef6-9052-874846b71cd2" path="/var/lib/kubelet/pods/9acd643c-f029-4ef6-9052-874846b71cd2/volumes" Nov 01 06:34:11.562997 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:34:11.562958 5040 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7125-857f5b\" is forbidden: User \"system:node:ip-10-0-143-195.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-195.us-west-1.compute.internal' and this object" podUID="9acd643c-f029-4ef6-9052-874846b71cd2" pod="opendatahub/test-curl-7125-857f5b" Nov 01 06:34:21.958466 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:21.958422 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-jxfb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 06:34:21.959306 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:21.959268 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" podUID="b6760e57-3c38-4dda-84e0-75b99b3a70a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 06:34:22.427691 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:22.427649 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pnbfj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 06:34:22.428024 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:22.427974 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" podUID="9e6f829a-4773-4740-a9e4-328091ee920f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 06:34:23.318574 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:23.318537 5040 scope.go:122] "RemoveContainer" containerID="61e90ae756846959eb12b6c79257e7f80ca03ebb46e3e02c4e94d0b6b7dfee96" Nov 01 06:34:27.010780 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:27.010741 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd99b3db-652hl"] Nov 01 06:34:27.011430 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:27.011415 5040 cpu_manager.go:414] "containerMap: removing container" podUID="9acd643c-f029-4ef6-9052-874846b71cd2" containerName="test-curl-7125-857f5b" Nov 01 06:34:27.011498 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:27.011432 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9acd643c-f029-4ef6-9052-874846b71cd2" containerName="test-curl-7125-857f5b" Nov 01 06:34:27.011586 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:27.011575 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="9acd643c-f029-4ef6-9052-874846b71cd2" containerName="test-curl-7125-857f5b" Nov 01 06:34:27.016382 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:27.016364 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd99b3db-652hl" Nov 01 06:34:27.068877 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:27.068840 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd99b3db-652hl"] Nov 01 06:34:27.114796 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:27.114763 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:34:27.114796 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:27.114763 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:34:27.115000 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:27.114808 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:34:27.115000 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:27.114820 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:34:27.138805 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:27.138771 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqkn4\" (UniqueName: \"kubernetes.io/projected/df21123c-bfab-43d6-ae4d-a5a44c762b6f-kube-api-access-fqkn4\") pod \"maas-api-revoke-sub-dd99b3db-652hl\" (UID: \"df21123c-bfab-43d6-ae4d-a5a44c762b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd99b3db-652hl" Nov 01 06:34:27.138805 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:27.138807 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df21123c-bfab-43d6-ae4d-a5a44c762b6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd99b3db-652hl\" (UID: \"df21123c-bfab-43d6-ae4d-a5a44c762b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd99b3db-652hl" Nov 01 06:34:27.139029 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:27.138962 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df21123c-bfab-43d6-ae4d-a5a44c762b6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd99b3db-652hl\" (UID: \"df21123c-bfab-43d6-ae4d-a5a44c762b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd99b3db-652hl" Nov 01 06:34:27.240144 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:27.240107 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqkn4\" (UniqueName: \"kubernetes.io/projected/df21123c-bfab-43d6-ae4d-a5a44c762b6f-kube-api-access-fqkn4\") pod \"maas-api-revoke-sub-dd99b3db-652hl\" (UID: \"df21123c-bfab-43d6-ae4d-a5a44c762b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd99b3db-652hl" Nov 01 06:34:27.240144 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:27.240153 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df21123c-bfab-43d6-ae4d-a5a44c762b6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd99b3db-652hl\" (UID: \"df21123c-bfab-43d6-ae4d-a5a44c762b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd99b3db-652hl" Nov 01 06:34:27.240433 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:27.240282 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df21123c-bfab-43d6-ae4d-a5a44c762b6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd99b3db-652hl\" (UID: \"df21123c-bfab-43d6-ae4d-a5a44c762b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd99b3db-652hl" Nov 01 06:34:27.240990 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:27.240962 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df21123c-bfab-43d6-ae4d-a5a44c762b6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd99b3db-652hl\" (UID: \"df21123c-bfab-43d6-ae4d-a5a44c762b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd99b3db-652hl" Nov 01 06:34:27.242744 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:27.242722 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df21123c-bfab-43d6-ae4d-a5a44c762b6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd99b3db-652hl\" (UID: \"df21123c-bfab-43d6-ae4d-a5a44c762b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd99b3db-652hl" Nov 01 06:34:27.301913 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:27.301834 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqkn4\" (UniqueName: \"kubernetes.io/projected/df21123c-bfab-43d6-ae4d-a5a44c762b6f-kube-api-access-fqkn4\") pod \"maas-api-revoke-sub-dd99b3db-652hl\" (UID: \"df21123c-bfab-43d6-ae4d-a5a44c762b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd99b3db-652hl" Nov 01 06:34:27.329681 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:27.329643 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd99b3db-652hl" Nov 01 06:34:27.708066 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:27.708042 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd99b3db-652hl"] Nov 01 06:34:27.709243 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:34:27.709212 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf21123c_bfab_43d6_ae4d_a5a44c762b6f.slice/crio-b4f762c9edfd304bc2c2659c2a0686d94434e7897b0e59b2741a68a0daf740a5 WatchSource:0}: Error finding container b4f762c9edfd304bc2c2659c2a0686d94434e7897b0e59b2741a68a0daf740a5: Status 404 returned error can't find the container with id b4f762c9edfd304bc2c2659c2a0686d94434e7897b0e59b2741a68a0daf740a5 Nov 01 06:34:27.960620 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:27.960529 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd99b3db-652hl" event={"ID":"df21123c-bfab-43d6-ae4d-a5a44c762b6f","Type":"ContainerStarted","Data":"b4f762c9edfd304bc2c2659c2a0686d94434e7897b0e59b2741a68a0daf740a5"} Nov 01 06:34:28.968357 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:28.968266 5040 generic.go:356] "Generic (PLEG): container finished" podID="df21123c-bfab-43d6-ae4d-a5a44c762b6f" containerID="55c5326605e256cf25308b0010cdbef212aabc95f722f22d379d4b2a1d96e7b0" exitCode=0 Nov 01 06:34:28.968762 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:28.968355 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd99b3db-652hl" event={"ID":"df21123c-bfab-43d6-ae4d-a5a44c762b6f","Type":"ContainerDied","Data":"55c5326605e256cf25308b0010cdbef212aabc95f722f22d379d4b2a1d96e7b0"} Nov 01 06:34:30.112454 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:30.112428 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd99b3db-652hl" Nov 01 06:34:30.274723 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:30.274692 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df21123c-bfab-43d6-ae4d-a5a44c762b6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df21123c-bfab-43d6-ae4d-a5a44c762b6f-maas-api-xks-ca\") pod \"df21123c-bfab-43d6-ae4d-a5a44c762b6f\" (UID: \"df21123c-bfab-43d6-ae4d-a5a44c762b6f\") " Nov 01 06:34:30.274912 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:30.274776 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df21123c-bfab-43d6-ae4d-a5a44c762b6f-kube-api-access-fqkn4\" (UniqueName: \"kubernetes.io/projected/df21123c-bfab-43d6-ae4d-a5a44c762b6f-kube-api-access-fqkn4\") pod \"df21123c-bfab-43d6-ae4d-a5a44c762b6f\" (UID: \"df21123c-bfab-43d6-ae4d-a5a44c762b6f\") " Nov 01 06:34:30.274912 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:30.274828 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df21123c-bfab-43d6-ae4d-a5a44c762b6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df21123c-bfab-43d6-ae4d-a5a44c762b6f-maas-api-openshift-ca\") pod \"df21123c-bfab-43d6-ae4d-a5a44c762b6f\" (UID: \"df21123c-bfab-43d6-ae4d-a5a44c762b6f\") " Nov 01 06:34:30.275254 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:30.275218 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df21123c-bfab-43d6-ae4d-a5a44c762b6f-maas-api-openshift-ca" pod "df21123c-bfab-43d6-ae4d-a5a44c762b6f" (UID: "df21123c-bfab-43d6-ae4d-a5a44c762b6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:34:30.275360 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:30.275342 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df21123c-bfab-43d6-ae4d-a5a44c762b6f-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:34:30.277006 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:30.276981 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df21123c-bfab-43d6-ae4d-a5a44c762b6f-maas-api-xks-ca" pod "df21123c-bfab-43d6-ae4d-a5a44c762b6f" (UID: "df21123c-bfab-43d6-ae4d-a5a44c762b6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:34:30.277108 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:30.277001 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df21123c-bfab-43d6-ae4d-a5a44c762b6f-kube-api-access-fqkn4" pod "df21123c-bfab-43d6-ae4d-a5a44c762b6f" (UID: "df21123c-bfab-43d6-ae4d-a5a44c762b6f"). InnerVolumeSpecName "kube-api-access-fqkn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:34:30.376684 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:30.376645 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df21123c-bfab-43d6-ae4d-a5a44c762b6f-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:34:30.376684 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:30.376683 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqkn4\" (UniqueName: \"kubernetes.io/projected/df21123c-bfab-43d6-ae4d-a5a44c762b6f-kube-api-access-fqkn4\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:34:30.983813 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:30.983787 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd99b3db-652hl" Nov 01 06:34:30.983813 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:30.983800 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd99b3db-652hl" event={"ID":"df21123c-bfab-43d6-ae4d-a5a44c762b6f","Type":"ContainerDied","Data":"b4f762c9edfd304bc2c2659c2a0686d94434e7897b0e59b2741a68a0daf740a5"} Nov 01 06:34:30.984032 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:30.983846 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4f762c9edfd304bc2c2659c2a0686d94434e7897b0e59b2741a68a0daf740a5" Nov 01 06:34:39.973016 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:34:39.972976 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:34:45.545661 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:45.545612 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" podUID="adbac784-4ded-4ca3-b985-3e59983fc637" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 06:34:49.643562 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:49.643517 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6"] Nov 01 06:34:49.644254 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:49.644226 5040 cpu_manager.go:414] "containerMap: removing container" podUID="df21123c-bfab-43d6-ae4d-a5a44c762b6f" containerName="revoke-keys" Nov 01 06:34:49.644306 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:49.644257 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df21123c-bfab-43d6-ae4d-a5a44c762b6f" containerName="revoke-keys" Nov 01 06:34:49.644457 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:49.644446 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="df21123c-bfab-43d6-ae4d-a5a44c762b6f" containerName="revoke-keys" Nov 01 06:34:49.648514 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:49.648492 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6" Nov 01 06:34:49.704662 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:49.704631 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6"] Nov 01 06:34:49.718138 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:49.718086 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8xbf\" (UniqueName: \"kubernetes.io/projected/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-kube-api-access-m8xbf\") pod \"autores-model-3680c5-kserve-756c7c6dfc-5nth6\" (UID: \"d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1\") " pod="ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6" Nov 01 06:34:49.718349 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:49.718166 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-tls-certs\") pod \"autores-model-3680c5-kserve-756c7c6dfc-5nth6\" (UID: \"d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1\") " pod="ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6" Nov 01 06:34:49.718349 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:49.718225 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-model-cache\") pod \"autores-model-3680c5-kserve-756c7c6dfc-5nth6\" (UID: \"d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1\") " pod="ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6" Nov 01 06:34:49.718349 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:49.718279 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-tmp-dir\") pod \"autores-model-3680c5-kserve-756c7c6dfc-5nth6\" (UID: \"d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1\") " pod="ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6" Nov 01 06:34:49.718349 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:49.718323 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-home\") pod \"autores-model-3680c5-kserve-756c7c6dfc-5nth6\" (UID: \"d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1\") " pod="ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6" Nov 01 06:34:49.718638 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:49.718362 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-dshm\") pod \"autores-model-3680c5-kserve-756c7c6dfc-5nth6\" (UID: \"d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1\") " pod="ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6" Nov 01 06:34:49.751213 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:49.751171 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-3680c5\"/\"autores-model-3680c5-kserve-self-signed-certs\"" Nov 01 06:34:49.751213 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:49.751190 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-3680c5\"/\"default-dockercfg-6df89\"" Nov 01 06:34:49.751588 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:49.751567 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-3680c5\"/\"openshift-service-ca.crt\"" Nov 01 06:34:49.751679 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:49.751584 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-3680c5\"/\"kube-root-ca.crt\"" Nov 01 06:34:49.820040 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:49.820005 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8xbf\" (UniqueName: \"kubernetes.io/projected/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-kube-api-access-m8xbf\") pod \"autores-model-3680c5-kserve-756c7c6dfc-5nth6\" (UID: \"d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1\") " pod="ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6" Nov 01 06:34:49.820040 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:49.820057 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-tls-certs\") pod \"autores-model-3680c5-kserve-756c7c6dfc-5nth6\" (UID: \"d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1\") " pod="ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6" Nov 01 06:34:49.820344 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:49.820098 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-model-cache\") pod \"autores-model-3680c5-kserve-756c7c6dfc-5nth6\" (UID: \"d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1\") " pod="ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6" Nov 01 06:34:49.820344 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:49.820133 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-tmp-dir\") pod \"autores-model-3680c5-kserve-756c7c6dfc-5nth6\" (UID: \"d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1\") " pod="ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6" Nov 01 06:34:49.820344 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:49.820167 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-home\") pod \"autores-model-3680c5-kserve-756c7c6dfc-5nth6\" (UID: \"d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1\") " pod="ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6" Nov 01 06:34:49.820344 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:49.820193 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-dshm\") pod \"autores-model-3680c5-kserve-756c7c6dfc-5nth6\" (UID: \"d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1\") " pod="ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6" Nov 01 06:34:49.820829 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:49.820801 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-model-cache\") pod \"autores-model-3680c5-kserve-756c7c6dfc-5nth6\" (UID: \"d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1\") " pod="ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6" Nov 01 06:34:49.820948 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:49.820925 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-tmp-dir\") pod \"autores-model-3680c5-kserve-756c7c6dfc-5nth6\" (UID: \"d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1\") " pod="ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6" Nov 01 06:34:49.821223 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:49.821193 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-home\") pod \"autores-model-3680c5-kserve-756c7c6dfc-5nth6\" (UID: \"d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1\") " pod="ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6" Nov 01 06:34:49.822736 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:49.822710 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-dshm\") pod \"autores-model-3680c5-kserve-756c7c6dfc-5nth6\" (UID: \"d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1\") " pod="ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6" Nov 01 06:34:49.822736 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:49.823292 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-tls-certs\") pod \"autores-model-3680c5-kserve-756c7c6dfc-5nth6\" (UID: \"d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1\") " pod="ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6" Nov 01 06:34:49.881648 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:49.881596 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8xbf\" (UniqueName: \"kubernetes.io/projected/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-kube-api-access-m8xbf\") pod \"autores-model-3680c5-kserve-756c7c6dfc-5nth6\" (UID: \"d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1\") " pod="ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6" Nov 01 06:34:49.962759 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:49.962668 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6" Nov 01 06:34:50.372119 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:50.368296 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6"] Nov 01 06:34:50.372119 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:34:50.371398 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6a4b60d_aa45_4d94_b6eb_f87602e5fbf1.slice/crio-bfd2714b186522e79c48bbb80ebdc6b5a524b2359a2d9b4197c06edbb6238dd0 WatchSource:0}: Error finding container bfd2714b186522e79c48bbb80ebdc6b5a524b2359a2d9b4197c06edbb6238dd0: Status 404 returned error can't find the container with id bfd2714b186522e79c48bbb80ebdc6b5a524b2359a2d9b4197c06edbb6238dd0 Nov 01 06:34:51.170903 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:51.170866 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6" event={"ID":"d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1","Type":"ContainerStarted","Data":"f6da7af3cfb487712c50a2c498b325a59aff0836cfcde0fe9d9cad8c9f8de4ed"} Nov 01 06:34:51.171311 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:51.170914 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6" event={"ID":"d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1","Type":"ContainerStarted","Data":"bfd2714b186522e79c48bbb80ebdc6b5a524b2359a2d9b4197c06edbb6238dd0"} Nov 01 06:34:51.318834 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:51.318780 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6" podStartSLOduration=2.318766763 podStartE2EDuration="2.318766763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:34:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:34:51.31681333 +0000 UTC m=+2969.971930498" watchObservedRunningTime="2026-11-01 06:34:51.318766763 +0000 UTC m=+2969.973883930" Nov 01 06:34:51.963367 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:51.963320 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6" Nov 01 06:34:59.962921 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:59.962862 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6" Nov 01 06:34:59.982867 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:34:59.982831 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6" Nov 01 06:35:00.241238 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:00.241138 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6" Nov 01 06:35:09.977638 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:35:09.977598 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:35:22.063512 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:22.063466 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 06:35:22.064063 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:22.063903 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 06:35:23.383567 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:23.383536 5040 scope.go:122] "RemoveContainer" containerID="6c22ce6cbbe3f48321a5db1cc3e917eb270257959be51bafc1ddc3890b875202" Nov 01 06:35:27.284624 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:27.284569 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" podUID="5daa03ca-824b-4799-a11b-4b3af1512c38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.123:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:35:27.315744 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:27.315701 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1989b21-tm5rg"] Nov 01 06:35:27.322237 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:27.322208 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1989b21-tm5rg" Nov 01 06:35:27.368681 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:27.368626 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1989b21-tm5rg"] Nov 01 06:35:27.397231 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:27.397193 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:35:27.411502 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:27.409924 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:35:27.411502 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:27.410034 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:35:27.411502 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:27.410090 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:35:27.434768 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:27.433206 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" podUID="1878b6fb-f8fe-4f4f-8f4e-056c66fc206a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:35:27.435705 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:27.435365 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv" podUID="72ab782a-6494-430b-a5eb-17f4b4f9ddd3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 06:35:27.461137 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:27.461098 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afe34fea-a723-41d1-958b-9adc0bf9d617-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1989b21-tm5rg\" (UID: \"afe34fea-a723-41d1-958b-9adc0bf9d617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1989b21-tm5rg" Nov 01 06:35:27.461463 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:27.461440 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whc25\" (UniqueName: \"kubernetes.io/projected/afe34fea-a723-41d1-958b-9adc0bf9d617-kube-api-access-whc25\") pod \"maas-api-revoke-sub-d1989b21-tm5rg\" (UID: \"afe34fea-a723-41d1-958b-9adc0bf9d617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1989b21-tm5rg" Nov 01 06:35:27.461730 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:27.461706 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afe34fea-a723-41d1-958b-9adc0bf9d617-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1989b21-tm5rg\" (UID: \"afe34fea-a723-41d1-958b-9adc0bf9d617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1989b21-tm5rg" Nov 01 06:35:27.563323 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:27.563226 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afe34fea-a723-41d1-958b-9adc0bf9d617-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1989b21-tm5rg\" (UID: \"afe34fea-a723-41d1-958b-9adc0bf9d617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1989b21-tm5rg" Nov 01 06:35:27.563323 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:27.563325 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afe34fea-a723-41d1-958b-9adc0bf9d617-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1989b21-tm5rg\" (UID: \"afe34fea-a723-41d1-958b-9adc0bf9d617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1989b21-tm5rg" Nov 01 06:35:27.563637 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:27.563358 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whc25\" (UniqueName: \"kubernetes.io/projected/afe34fea-a723-41d1-958b-9adc0bf9d617-kube-api-access-whc25\") pod \"maas-api-revoke-sub-d1989b21-tm5rg\" (UID: \"afe34fea-a723-41d1-958b-9adc0bf9d617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1989b21-tm5rg" Nov 01 06:35:27.564050 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:27.564029 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afe34fea-a723-41d1-958b-9adc0bf9d617-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1989b21-tm5rg\" (UID: \"afe34fea-a723-41d1-958b-9adc0bf9d617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1989b21-tm5rg" Nov 01 06:35:27.565780 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:27.565755 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afe34fea-a723-41d1-958b-9adc0bf9d617-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1989b21-tm5rg\" (UID: \"afe34fea-a723-41d1-958b-9adc0bf9d617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1989b21-tm5rg" Nov 01 06:35:27.624747 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:27.624717 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whc25\" (UniqueName: \"kubernetes.io/projected/afe34fea-a723-41d1-958b-9adc0bf9d617-kube-api-access-whc25\") pod \"maas-api-revoke-sub-d1989b21-tm5rg\" (UID: \"afe34fea-a723-41d1-958b-9adc0bf9d617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1989b21-tm5rg" Nov 01 06:35:27.648541 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:27.648502 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1989b21-tm5rg" Nov 01 06:35:28.026087 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:28.026054 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1989b21-tm5rg"] Nov 01 06:35:28.027697 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:35:28.027670 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe34fea_a723_41d1_958b_9adc0bf9d617.slice/crio-0904e1ceb0912a0cae920db65786930f1ee1a98230bbf3f7fa8c3518b11bb147 WatchSource:0}: Error finding container 0904e1ceb0912a0cae920db65786930f1ee1a98230bbf3f7fa8c3518b11bb147: Status 404 returned error can't find the container with id 0904e1ceb0912a0cae920db65786930f1ee1a98230bbf3f7fa8c3518b11bb147 Nov 01 06:35:28.512932 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:28.512896 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1989b21-tm5rg" event={"ID":"afe34fea-a723-41d1-958b-9adc0bf9d617","Type":"ContainerStarted","Data":"787fa3e25940ef0e4f9292146e27c8a54bec22bd95778e30f069ba8eef2190e3"} Nov 01 06:35:28.513363 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:28.512952 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1989b21-tm5rg" event={"ID":"afe34fea-a723-41d1-958b-9adc0bf9d617","Type":"ContainerStarted","Data":"0904e1ceb0912a0cae920db65786930f1ee1a98230bbf3f7fa8c3518b11bb147"} Nov 01 06:35:28.631915 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:28.631810 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1989b21-tm5rg" podStartSLOduration=1.631796461 podStartE2EDuration="1.631796461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:35:28.629402827 +0000 UTC m=+3007.284519996" watchObservedRunningTime="2026-11-01 06:35:28.631796461 +0000 UTC m=+3007.286913657" Nov 01 06:35:29.524608 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:29.524569 5040 generic.go:356] "Generic (PLEG): container finished" podID="afe34fea-a723-41d1-958b-9adc0bf9d617" containerID="787fa3e25940ef0e4f9292146e27c8a54bec22bd95778e30f069ba8eef2190e3" exitCode=0 Nov 01 06:35:29.525124 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:29.524646 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1989b21-tm5rg" event={"ID":"afe34fea-a723-41d1-958b-9adc0bf9d617","Type":"ContainerDied","Data":"787fa3e25940ef0e4f9292146e27c8a54bec22bd95778e30f069ba8eef2190e3"} Nov 01 06:35:30.690133 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:30.690104 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1989b21-tm5rg" Nov 01 06:35:30.800183 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:30.800084 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afe34fea-a723-41d1-958b-9adc0bf9d617-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afe34fea-a723-41d1-958b-9adc0bf9d617-maas-api-xks-ca\") pod \"afe34fea-a723-41d1-958b-9adc0bf9d617\" (UID: \"afe34fea-a723-41d1-958b-9adc0bf9d617\") " Nov 01 06:35:30.800383 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:30.800189 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afe34fea-a723-41d1-958b-9adc0bf9d617-kube-api-access-whc25\" (UniqueName: \"kubernetes.io/projected/afe34fea-a723-41d1-958b-9adc0bf9d617-kube-api-access-whc25\") pod \"afe34fea-a723-41d1-958b-9adc0bf9d617\" (UID: \"afe34fea-a723-41d1-958b-9adc0bf9d617\") " Nov 01 06:35:30.800383 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:30.800285 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afe34fea-a723-41d1-958b-9adc0bf9d617-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afe34fea-a723-41d1-958b-9adc0bf9d617-maas-api-openshift-ca\") pod \"afe34fea-a723-41d1-958b-9adc0bf9d617\" (UID: \"afe34fea-a723-41d1-958b-9adc0bf9d617\") " Nov 01 06:35:30.800660 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:30.800635 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afe34fea-a723-41d1-958b-9adc0bf9d617-maas-api-openshift-ca" pod "afe34fea-a723-41d1-958b-9adc0bf9d617" (UID: "afe34fea-a723-41d1-958b-9adc0bf9d617"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:35:30.802334 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:30.802310 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afe34fea-a723-41d1-958b-9adc0bf9d617-maas-api-xks-ca" pod "afe34fea-a723-41d1-958b-9adc0bf9d617" (UID: "afe34fea-a723-41d1-958b-9adc0bf9d617"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:35:30.802403 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:30.802383 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe34fea-a723-41d1-958b-9adc0bf9d617-kube-api-access-whc25" pod "afe34fea-a723-41d1-958b-9adc0bf9d617" (UID: "afe34fea-a723-41d1-958b-9adc0bf9d617"). InnerVolumeSpecName "kube-api-access-whc25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:35:30.902043 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:30.902008 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afe34fea-a723-41d1-958b-9adc0bf9d617-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:35:30.902043 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:30.902040 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afe34fea-a723-41d1-958b-9adc0bf9d617-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:35:30.902043 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:30.902052 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whc25\" (UniqueName: \"kubernetes.io/projected/afe34fea-a723-41d1-958b-9adc0bf9d617-kube-api-access-whc25\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:35:31.539137 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:31.539107 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1989b21-tm5rg" Nov 01 06:35:31.539137 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:31.539127 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1989b21-tm5rg" event={"ID":"afe34fea-a723-41d1-958b-9adc0bf9d617","Type":"ContainerDied","Data":"0904e1ceb0912a0cae920db65786930f1ee1a98230bbf3f7fa8c3518b11bb147"} Nov 01 06:35:31.539339 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:31.539173 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0904e1ceb0912a0cae920db65786930f1ee1a98230bbf3f7fa8c3518b11bb147" Nov 01 06:35:36.662840 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:36.662804 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-qtnvw"] Nov 01 06:35:36.712246 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:36.712206 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-qtnvw"] Nov 01 06:35:37.515115 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:35:37.515079 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cdf9878-866f-49b0-b6a6-d59596787b11" path="/var/lib/kubelet/pods/7cdf9878-866f-49b0-b6a6-d59596787b11/volumes" Nov 01 06:35:39.976691 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:35:39.976646 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:36:09.005197 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:09.005159 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6"] Nov 01 06:36:09.973324 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:36:09.973282 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:36:24.027770 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:24.027726 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6" podUID="d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1" containerName="main" containerID="cri-o://f6da7af3cfb487712c50a2c498b325a59aff0836cfcde0fe9d9cad8c9f8de4ed" gracePeriod=45 Nov 01 06:36:24.296039 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:24.295974 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6" Nov 01 06:36:24.428907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:24.428872 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-tmp-dir\") pod \"d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1\" (UID: \"d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1\") " Nov 01 06:36:24.429096 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:24.428938 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-dshm\") pod \"d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1\" (UID: \"d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1\") " Nov 01 06:36:24.429096 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:24.429077 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-tls-certs\" (UniqueName: \"kubernetes.io/secret/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-tls-certs\") pod \"d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1\" (UID: \"d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1\") " Nov 01 06:36:24.429212 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:24.429134 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-kube-api-access-m8xbf\" (UniqueName: \"kubernetes.io/projected/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-kube-api-access-m8xbf\") pod \"d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1\" (UID: \"d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1\") " Nov 01 06:36:24.429212 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:24.429189 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-tmp-dir" pod "d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1" (UID: "d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:36:24.429337 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:24.429246 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-model-cache\") pod \"d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1\" (UID: \"d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1\") " Nov 01 06:36:24.429337 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:24.429320 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-home\" (UniqueName: \"kubernetes.io/empty-dir/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-home\") pod \"d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1\" (UID: \"d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1\") " Nov 01 06:36:24.430049 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:24.429705 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-model-cache" pod "d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1" (UID: "d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:36:24.430049 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:24.429929 5040 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-model-cache\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:24.430049 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:24.429931 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-home" pod "d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1" (UID: "d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:36:24.430049 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:24.429951 5040 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-tmp-dir\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:24.431528 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:24.431472 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-dshm" pod "d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1" (UID: "d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:36:24.431693 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:24.431669 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-kube-api-access-m8xbf" pod "d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1" (UID: "d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1"). InnerVolumeSpecName "kube-api-access-m8xbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:36:24.432987 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:24.432955 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-tls-certs" pod "d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1" (UID: "d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:36:24.531169 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:24.531133 5040 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-dshm\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:24.531169 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:24.531162 5040 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-tls-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:24.531169 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:24.531172 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8xbf\" (UniqueName: \"kubernetes.io/projected/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-kube-api-access-m8xbf\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:24.531471 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:24.531181 5040 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1-home\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:24.928128 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:24.928086 5040 generic.go:356] "Generic (PLEG): container finished" podID="d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1" containerID="f6da7af3cfb487712c50a2c498b325a59aff0836cfcde0fe9d9cad8c9f8de4ed" exitCode=0 Nov 01 06:36:24.928299 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:24.928133 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6" Nov 01 06:36:24.928299 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:24.928163 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6" event={"ID":"d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1","Type":"ContainerDied","Data":"f6da7af3cfb487712c50a2c498b325a59aff0836cfcde0fe9d9cad8c9f8de4ed"} Nov 01 06:36:24.928299 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:24.928207 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6" event={"ID":"d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1","Type":"ContainerDied","Data":"bfd2714b186522e79c48bbb80ebdc6b5a524b2359a2d9b4197c06edbb6238dd0"} Nov 01 06:36:24.928299 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:24.928226 5040 scope.go:122] "RemoveContainer" containerID="f6da7af3cfb487712c50a2c498b325a59aff0836cfcde0fe9d9cad8c9f8de4ed" Nov 01 06:36:24.940205 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:24.940183 5040 scope.go:122] "RemoveContainer" containerID="f6da7af3cfb487712c50a2c498b325a59aff0836cfcde0fe9d9cad8c9f8de4ed" Nov 01 06:36:24.940533 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:36:24.940508 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6da7af3cfb487712c50a2c498b325a59aff0836cfcde0fe9d9cad8c9f8de4ed\": container with ID starting with f6da7af3cfb487712c50a2c498b325a59aff0836cfcde0fe9d9cad8c9f8de4ed not found: ID does not exist" containerID="f6da7af3cfb487712c50a2c498b325a59aff0836cfcde0fe9d9cad8c9f8de4ed" Nov 01 06:36:24.940647 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:24.940540 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6da7af3cfb487712c50a2c498b325a59aff0836cfcde0fe9d9cad8c9f8de4ed"} err="failed to get container status \"f6da7af3cfb487712c50a2c498b325a59aff0836cfcde0fe9d9cad8c9f8de4ed\": rpc error: code = NotFound desc = could not find container \"f6da7af3cfb487712c50a2c498b325a59aff0836cfcde0fe9d9cad8c9f8de4ed\": container with ID starting with f6da7af3cfb487712c50a2c498b325a59aff0836cfcde0fe9d9cad8c9f8de4ed not found: ID does not exist" Nov 01 06:36:25.102823 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:25.102782 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6"] Nov 01 06:36:25.151333 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:25.151286 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6"] Nov 01 06:36:25.230528 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:25.230404 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-3680c5/autores-model-3680c5-kserve-756c7c6dfc-5nth6" podUID="d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.203:8000/ready\": context deadline exceeded" Nov 01 06:36:25.516555 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:25.516522 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1" path="/var/lib/kubelet/pods/d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1/volumes" Nov 01 06:36:39.973518 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:36:39.973452 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:36:59.880986 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:59.880952 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp"] Nov 01 06:36:59.881451 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:59.881402 5040 cpu_manager.go:414] "containerMap: removing container" podUID="d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1" containerName="main" Nov 01 06:36:59.881451 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:59.881413 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1" containerName="main" Nov 01 06:36:59.881451 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:59.881447 5040 cpu_manager.go:414] "containerMap: removing container" podUID="afe34fea-a723-41d1-958b-9adc0bf9d617" containerName="revoke-keys" Nov 01 06:36:59.881451 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:59.881453 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afe34fea-a723-41d1-958b-9adc0bf9d617" containerName="revoke-keys" Nov 01 06:36:59.881622 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:59.881567 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="afe34fea-a723-41d1-958b-9adc0bf9d617" containerName="revoke-keys" Nov 01 06:36:59.881622 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:59.881580 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6a4b60d-aa45-4d94-b6eb-f87602e5fbf1" containerName="main" Nov 01 06:36:59.884903 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:59.884884 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp" Nov 01 06:36:59.938882 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:59.938850 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-747b7b\"/\"kube-root-ca.crt\"" Nov 01 06:36:59.989055 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:59.989017 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-747b7b\"/\"openshift-service-ca.crt\"" Nov 01 06:36:59.989259 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:59.989239 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-747b7b\"/\"default-dockercfg-m7px6\"" Nov 01 06:36:59.989310 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:59.989247 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-747b7b\"/\"rate-test-model-747b7b-kserve-self-signed-certs\"" Nov 01 06:36:59.992339 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:36:59.992318 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp"] Nov 01 06:37:00.023007 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:00.022979 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-dshm\") pod \"rate-test-model-747b7b-kserve-d645d897c-p8qvp\" (UID: \"8acef8ba-1ac8-4cfb-b853-bb9a63afe24e\") " pod="ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp" Nov 01 06:37:00.023163 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:00.023014 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-tmp-dir\") pod \"rate-test-model-747b7b-kserve-d645d897c-p8qvp\" (UID: \"8acef8ba-1ac8-4cfb-b853-bb9a63afe24e\") " pod="ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp" Nov 01 06:37:00.023163 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:00.023035 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhjzg\" (UniqueName: \"kubernetes.io/projected/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-kube-api-access-qhjzg\") pod \"rate-test-model-747b7b-kserve-d645d897c-p8qvp\" (UID: \"8acef8ba-1ac8-4cfb-b853-bb9a63afe24e\") " pod="ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp" Nov 01 06:37:00.023163 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:00.023058 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-model-cache\") pod \"rate-test-model-747b7b-kserve-d645d897c-p8qvp\" (UID: \"8acef8ba-1ac8-4cfb-b853-bb9a63afe24e\") " pod="ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp" Nov 01 06:37:00.023163 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:00.023159 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-tls-certs\") pod \"rate-test-model-747b7b-kserve-d645d897c-p8qvp\" (UID: \"8acef8ba-1ac8-4cfb-b853-bb9a63afe24e\") " pod="ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp" Nov 01 06:37:00.023308 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:00.023268 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-home\") pod \"rate-test-model-747b7b-kserve-d645d897c-p8qvp\" (UID: \"8acef8ba-1ac8-4cfb-b853-bb9a63afe24e\") " pod="ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp" Nov 01 06:37:00.124661 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:00.124612 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-dshm\") pod \"rate-test-model-747b7b-kserve-d645d897c-p8qvp\" (UID: \"8acef8ba-1ac8-4cfb-b853-bb9a63afe24e\") " pod="ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp" Nov 01 06:37:00.124661 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:00.124659 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-tmp-dir\") pod \"rate-test-model-747b7b-kserve-d645d897c-p8qvp\" (UID: \"8acef8ba-1ac8-4cfb-b853-bb9a63afe24e\") " pod="ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp" Nov 01 06:37:00.124939 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:00.124677 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhjzg\" (UniqueName: \"kubernetes.io/projected/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-kube-api-access-qhjzg\") pod \"rate-test-model-747b7b-kserve-d645d897c-p8qvp\" (UID: \"8acef8ba-1ac8-4cfb-b853-bb9a63afe24e\") " pod="ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp" Nov 01 06:37:00.124939 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:00.124699 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-model-cache\") pod \"rate-test-model-747b7b-kserve-d645d897c-p8qvp\" (UID: \"8acef8ba-1ac8-4cfb-b853-bb9a63afe24e\") " pod="ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp" Nov 01 06:37:00.124939 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:00.124746 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-tls-certs\") pod \"rate-test-model-747b7b-kserve-d645d897c-p8qvp\" (UID: \"8acef8ba-1ac8-4cfb-b853-bb9a63afe24e\") " pod="ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp" Nov 01 06:37:00.124939 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:00.124830 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-home\") pod \"rate-test-model-747b7b-kserve-d645d897c-p8qvp\" (UID: \"8acef8ba-1ac8-4cfb-b853-bb9a63afe24e\") " pod="ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp" Nov 01 06:37:00.125211 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:00.125183 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-tmp-dir\") pod \"rate-test-model-747b7b-kserve-d645d897c-p8qvp\" (UID: \"8acef8ba-1ac8-4cfb-b853-bb9a63afe24e\") " pod="ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp" Nov 01 06:37:00.125273 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:00.125210 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-model-cache\") pod \"rate-test-model-747b7b-kserve-d645d897c-p8qvp\" (UID: \"8acef8ba-1ac8-4cfb-b853-bb9a63afe24e\") " pod="ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp" Nov 01 06:37:00.125572 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:00.125549 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-home\") pod \"rate-test-model-747b7b-kserve-d645d897c-p8qvp\" (UID: \"8acef8ba-1ac8-4cfb-b853-bb9a63afe24e\") " pod="ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp" Nov 01 06:37:00.127567 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:00.127541 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-tls-certs\") pod \"rate-test-model-747b7b-kserve-d645d897c-p8qvp\" (UID: \"8acef8ba-1ac8-4cfb-b853-bb9a63afe24e\") " pod="ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp" Nov 01 06:37:00.127705 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:00.127669 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-dshm\") pod \"rate-test-model-747b7b-kserve-d645d897c-p8qvp\" (UID: \"8acef8ba-1ac8-4cfb-b853-bb9a63afe24e\") " pod="ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp" Nov 01 06:37:00.193239 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:00.193151 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhjzg\" (UniqueName: \"kubernetes.io/projected/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-kube-api-access-qhjzg\") pod \"rate-test-model-747b7b-kserve-d645d897c-p8qvp\" (UID: \"8acef8ba-1ac8-4cfb-b853-bb9a63afe24e\") " pod="ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp" Nov 01 06:37:00.196841 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:00.196811 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp" Nov 01 06:37:00.584145 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:00.584113 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp"] Nov 01 06:37:00.585262 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:37:00.585234 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8acef8ba_1ac8_4cfb_b853_bb9a63afe24e.slice/crio-08ea117b8cdb50bf07e8c1366450f16ed868f819699dc103a33f9e79cfb0f62a WatchSource:0}: Error finding container 08ea117b8cdb50bf07e8c1366450f16ed868f819699dc103a33f9e79cfb0f62a: Status 404 returned error can't find the container with id 08ea117b8cdb50bf07e8c1366450f16ed868f819699dc103a33f9e79cfb0f62a Nov 01 06:37:01.160337 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:01.160301 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp" event={"ID":"8acef8ba-1ac8-4cfb-b853-bb9a63afe24e","Type":"ContainerStarted","Data":"b38b6aacefa521f04d6e8bbbad1cdadef636ed1eed3527d1787fdc2cad049243"} Nov 01 06:37:01.160758 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:01.160353 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp" event={"ID":"8acef8ba-1ac8-4cfb-b853-bb9a63afe24e","Type":"ContainerStarted","Data":"08ea117b8cdb50bf07e8c1366450f16ed868f819699dc103a33f9e79cfb0f62a"} Nov 01 06:37:01.197781 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:01.197743 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp" Nov 01 06:37:01.284518 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:01.283794 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp" podStartSLOduration=2.283776251 podStartE2EDuration="2.283776251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:36:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:37:01.279205005 +0000 UTC m=+3099.934322172" watchObservedRunningTime="2026-11-01 06:37:01.283776251 +0000 UTC m=+3099.938893418" Nov 01 06:37:09.974644 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:37:09.974606 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:37:10.198049 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:10.198014 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp" Nov 01 06:37:10.210773 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:10.210749 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp" Nov 01 06:37:10.248984 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:10.248955 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp" Nov 01 06:37:39.973531 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:37:39.973470 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:37:47.649657 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.649613 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn"] Nov 01 06:37:47.654983 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.654963 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" Nov 01 06:37:47.699354 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.699319 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh"] Nov 01 06:37:47.703159 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.703142 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" Nov 01 06:37:47.707196 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.707170 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-istiod-ca-cert\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn\" (UID: \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" Nov 01 06:37:47.707322 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.707208 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhk2s\" (UniqueName: \"kubernetes.io/projected/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-kube-api-access-rhk2s\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn\" (UID: \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" Nov 01 06:37:47.707404 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.707382 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-istio-data\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn\" (UID: \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" Nov 01 06:37:47.707448 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.707423 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-istio-token\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn\" (UID: \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" Nov 01 06:37:47.707536 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.707515 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-istio-envoy\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn\" (UID: \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" Nov 01 06:37:47.707660 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.707641 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-istio-podinfo\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn\" (UID: \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" Nov 01 06:37:47.707707 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.707668 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-credential-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn\" (UID: \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" Nov 01 06:37:47.708138 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.708097 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-workload-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn\" (UID: \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" Nov 01 06:37:47.708612 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.708523 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-workload-certs\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn\" (UID: \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" Nov 01 06:37:47.774626 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.774592 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn"] Nov 01 06:37:47.791471 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.791437 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh"] Nov 01 06:37:47.809851 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.809823 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/952eafb2-b8bc-457d-9bd6-4560f6176b74-credential-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh\" (UID: \"952eafb2-b8bc-457d-9bd6-4560f6176b74\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" Nov 01 06:37:47.810016 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.809855 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/952eafb2-b8bc-457d-9bd6-4560f6176b74-istio-podinfo\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh\" (UID: \"952eafb2-b8bc-457d-9bd6-4560f6176b74\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" Nov 01 06:37:47.810016 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.809980 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-istio-data\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn\" (UID: \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" Nov 01 06:37:47.810092 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.810040 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-istio-token\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn\" (UID: \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" Nov 01 06:37:47.810092 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.810077 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-istio-envoy\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn\" (UID: \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" Nov 01 06:37:47.810238 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.810216 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkh5t\" (UniqueName: \"kubernetes.io/projected/952eafb2-b8bc-457d-9bd6-4560f6176b74-kube-api-access-xkh5t\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh\" (UID: \"952eafb2-b8bc-457d-9bd6-4560f6176b74\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" Nov 01 06:37:47.810305 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.810274 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/952eafb2-b8bc-457d-9bd6-4560f6176b74-workload-certs\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh\" (UID: \"952eafb2-b8bc-457d-9bd6-4560f6176b74\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" Nov 01 06:37:47.810405 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.810378 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-istio-podinfo\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn\" (UID: \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" Nov 01 06:37:47.810457 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.810406 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-istio-data\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn\" (UID: \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" Nov 01 06:37:47.810457 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.810424 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-credential-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn\" (UID: \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" Nov 01 06:37:47.810609 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.810507 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/952eafb2-b8bc-457d-9bd6-4560f6176b74-istio-token\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh\" (UID: \"952eafb2-b8bc-457d-9bd6-4560f6176b74\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" Nov 01 06:37:47.810609 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.810541 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/952eafb2-b8bc-457d-9bd6-4560f6176b74-istio-data\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh\" (UID: \"952eafb2-b8bc-457d-9bd6-4560f6176b74\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" Nov 01 06:37:47.810758 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.810627 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-workload-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn\" (UID: \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" Nov 01 06:37:47.810758 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.810672 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-workload-certs\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn\" (UID: \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" Nov 01 06:37:47.810758 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.810701 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/952eafb2-b8bc-457d-9bd6-4560f6176b74-istio-envoy\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh\" (UID: \"952eafb2-b8bc-457d-9bd6-4560f6176b74\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" Nov 01 06:37:47.810758 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.810739 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-istiod-ca-cert\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn\" (UID: \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" Nov 01 06:37:47.811042 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.810767 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-credential-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn\" (UID: \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" Nov 01 06:37:47.811042 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.810773 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhk2s\" (UniqueName: \"kubernetes.io/projected/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-kube-api-access-rhk2s\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn\" (UID: \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" Nov 01 06:37:47.811042 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.810863 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/952eafb2-b8bc-457d-9bd6-4560f6176b74-workload-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh\" (UID: \"952eafb2-b8bc-457d-9bd6-4560f6176b74\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" Nov 01 06:37:47.811042 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.810889 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/952eafb2-b8bc-457d-9bd6-4560f6176b74-istiod-ca-cert\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh\" (UID: \"952eafb2-b8bc-457d-9bd6-4560f6176b74\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" Nov 01 06:37:47.811277 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.811068 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-workload-certs\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn\" (UID: \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" Nov 01 06:37:47.811340 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.811312 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-workload-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn\" (UID: \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" Nov 01 06:37:47.811544 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.811519 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-istiod-ca-cert\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn\" (UID: \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" Nov 01 06:37:47.812605 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.812585 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-istio-envoy\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn\" (UID: \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" Nov 01 06:37:47.813292 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.813276 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-istio-podinfo\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn\" (UID: \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" Nov 01 06:37:47.870940 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.870905 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-istio-token\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn\" (UID: \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" Nov 01 06:37:47.871873 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.871841 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhk2s\" (UniqueName: \"kubernetes.io/projected/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-kube-api-access-rhk2s\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn\" (UID: \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" Nov 01 06:37:47.912384 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.912312 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkh5t\" (UniqueName: \"kubernetes.io/projected/952eafb2-b8bc-457d-9bd6-4560f6176b74-kube-api-access-xkh5t\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh\" (UID: \"952eafb2-b8bc-457d-9bd6-4560f6176b74\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" Nov 01 06:37:47.912384 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.912354 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/952eafb2-b8bc-457d-9bd6-4560f6176b74-workload-certs\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh\" (UID: \"952eafb2-b8bc-457d-9bd6-4560f6176b74\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" Nov 01 06:37:47.912646 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.912400 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/952eafb2-b8bc-457d-9bd6-4560f6176b74-istio-token\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh\" (UID: \"952eafb2-b8bc-457d-9bd6-4560f6176b74\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" Nov 01 06:37:47.912646 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.912420 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/952eafb2-b8bc-457d-9bd6-4560f6176b74-istio-data\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh\" (UID: \"952eafb2-b8bc-457d-9bd6-4560f6176b74\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" Nov 01 06:37:47.912646 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.912451 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/952eafb2-b8bc-457d-9bd6-4560f6176b74-istio-envoy\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh\" (UID: \"952eafb2-b8bc-457d-9bd6-4560f6176b74\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" Nov 01 06:37:47.912646 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.912504 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/952eafb2-b8bc-457d-9bd6-4560f6176b74-workload-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh\" (UID: \"952eafb2-b8bc-457d-9bd6-4560f6176b74\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" Nov 01 06:37:47.912646 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.912529 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/952eafb2-b8bc-457d-9bd6-4560f6176b74-istiod-ca-cert\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh\" (UID: \"952eafb2-b8bc-457d-9bd6-4560f6176b74\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" Nov 01 06:37:47.912949 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.912923 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/952eafb2-b8bc-457d-9bd6-4560f6176b74-credential-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh\" (UID: \"952eafb2-b8bc-457d-9bd6-4560f6176b74\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" Nov 01 06:37:47.913090 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.912964 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/952eafb2-b8bc-457d-9bd6-4560f6176b74-istio-podinfo\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh\" (UID: \"952eafb2-b8bc-457d-9bd6-4560f6176b74\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" Nov 01 06:37:47.913090 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.912928 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/952eafb2-b8bc-457d-9bd6-4560f6176b74-istio-data\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh\" (UID: \"952eafb2-b8bc-457d-9bd6-4560f6176b74\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" Nov 01 06:37:47.913090 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.912824 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/952eafb2-b8bc-457d-9bd6-4560f6176b74-workload-certs\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh\" (UID: \"952eafb2-b8bc-457d-9bd6-4560f6176b74\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" Nov 01 06:37:47.913230 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.913167 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/952eafb2-b8bc-457d-9bd6-4560f6176b74-workload-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh\" (UID: \"952eafb2-b8bc-457d-9bd6-4560f6176b74\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" Nov 01 06:37:47.913230 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.913192 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/952eafb2-b8bc-457d-9bd6-4560f6176b74-credential-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh\" (UID: \"952eafb2-b8bc-457d-9bd6-4560f6176b74\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" Nov 01 06:37:47.913330 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.913305 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/952eafb2-b8bc-457d-9bd6-4560f6176b74-istiod-ca-cert\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh\" (UID: \"952eafb2-b8bc-457d-9bd6-4560f6176b74\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" Nov 01 06:37:47.914731 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.914712 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/952eafb2-b8bc-457d-9bd6-4560f6176b74-istio-envoy\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh\" (UID: \"952eafb2-b8bc-457d-9bd6-4560f6176b74\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" Nov 01 06:37:47.915244 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.915226 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/952eafb2-b8bc-457d-9bd6-4560f6176b74-istio-podinfo\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh\" (UID: \"952eafb2-b8bc-457d-9bd6-4560f6176b74\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" Nov 01 06:37:47.968272 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.968228 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" Nov 01 06:37:47.973991 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.973960 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/952eafb2-b8bc-457d-9bd6-4560f6176b74-istio-token\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh\" (UID: \"952eafb2-b8bc-457d-9bd6-4560f6176b74\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" Nov 01 06:37:47.974089 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:47.974029 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkh5t\" (UniqueName: \"kubernetes.io/projected/952eafb2-b8bc-457d-9bd6-4560f6176b74-kube-api-access-xkh5t\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh\" (UID: \"952eafb2-b8bc-457d-9bd6-4560f6176b74\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" Nov 01 06:37:48.027044 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:48.027014 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" Nov 01 06:37:48.398929 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:37:48.398875 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod850bc9c2_fdcd_4a05_b7dc_1f04ea59f821.slice/crio-26b1bd34e621ceec582128e263928ea38227b664d2d88a38f5d14849729b4687 WatchSource:0}: Error finding container 26b1bd34e621ceec582128e263928ea38227b664d2d88a38f5d14849729b4687: Status 404 returned error can't find the container with id 26b1bd34e621ceec582128e263928ea38227b664d2d88a38f5d14849729b4687 Nov 01 06:37:48.399555 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:48.399521 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn"] Nov 01 06:37:48.406131 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:48.406093 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:37:48.406219 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:48.406177 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:37:48.406262 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:48.406218 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:37:48.453233 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:48.453209 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh"] Nov 01 06:37:48.456619 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:37:48.456591 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod952eafb2_b8bc_457d_9bd6_4560f6176b74.slice/crio-53935ec2522d5353a0beba50ce3c68ad578cc69c15135de296bedeb1d9a7cd47 WatchSource:0}: Error finding container 53935ec2522d5353a0beba50ce3c68ad578cc69c15135de296bedeb1d9a7cd47: Status 404 returned error can't find the container with id 53935ec2522d5353a0beba50ce3c68ad578cc69c15135de296bedeb1d9a7cd47 Nov 01 06:37:48.459516 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:48.459447 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:37:48.459608 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:48.459565 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:37:48.459647 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:48.459607 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:37:48.506809 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:48.506775 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" event={"ID":"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821","Type":"ContainerStarted","Data":"812ea0d4f5cfd3b440e1553c372b5a8cf350d484522a4897272377d1cde51d08"} Nov 01 06:37:48.507017 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:48.507002 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" event={"ID":"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821","Type":"ContainerStarted","Data":"26b1bd34e621ceec582128e263928ea38227b664d2d88a38f5d14849729b4687"} Nov 01 06:37:48.510877 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:48.510717 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" event={"ID":"952eafb2-b8bc-457d-9bd6-4560f6176b74","Type":"ContainerStarted","Data":"53935ec2522d5353a0beba50ce3c68ad578cc69c15135de296bedeb1d9a7cd47"} Nov 01 06:37:48.636791 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:48.636738 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" podStartSLOduration=1.636724585 podStartE2EDuration="1.636724585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:37:48.632202497 +0000 UTC m=+3147.287319667" watchObservedRunningTime="2026-11-01 06:37:48.636724585 +0000 UTC m=+3147.291841820" Nov 01 06:37:48.969169 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:48.969126 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" Nov 01 06:37:49.560171 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:49.560112 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" event={"ID":"952eafb2-b8bc-457d-9bd6-4560f6176b74","Type":"ContainerStarted","Data":"fbbde0dd042ee7e3ed0d145a5bf4416e5e264307bedf4b951e88a75bb70c7a94"} Nov 01 06:37:49.749119 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:49.749060 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" podStartSLOduration=2.749041648 podStartE2EDuration="2.749041648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:37:49.744874148 +0000 UTC m=+3148.399991315" watchObservedRunningTime="2026-11-01 06:37:49.749041648 +0000 UTC m=+3148.404158815" Nov 01 06:37:49.970352 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:49.970260 5040 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 01 06:37:49.970352 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:49.970322 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" podUID="850bc9c2-fdcd-4a05-b7dc-1f04ea59f821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 01 06:37:50.031302 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:50.031265 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" Nov 01 06:37:50.031725 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:50.031699 5040 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 01 06:37:50.031910 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:50.031886 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" podUID="952eafb2-b8bc-457d-9bd6-4560f6176b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 01 06:37:50.968703 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:50.968669 5040 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 01 06:37:50.968888 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:50.968732 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" podUID="850bc9c2-fdcd-4a05-b7dc-1f04ea59f821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 01 06:37:51.030835 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:51.030798 5040 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 01 06:37:51.031426 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:51.031391 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" podUID="952eafb2-b8bc-457d-9bd6-4560f6176b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 01 06:37:51.679295 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:51.679259 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82"] Nov 01 06:37:51.679555 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:51.679515 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" podUID="d9577376-97a1-4915-b69e-26e60c29fce4" containerName="istio-proxy" containerID="cri-o://c1beeaff29df2cac39f16cd34c4fa153761bee8150cb413aa013d910af704a19" gracePeriod=30 Nov 01 06:37:51.968794 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:51.968709 5040 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 01 06:37:51.968794 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:51.968771 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" podUID="850bc9c2-fdcd-4a05-b7dc-1f04ea59f821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 01 06:37:52.028905 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:52.028870 5040 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 01 06:37:52.029199 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:52.029167 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" podUID="952eafb2-b8bc-457d-9bd6-4560f6176b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 01 06:37:52.969397 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:52.969355 5040 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 01 06:37:52.969924 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:52.969411 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" podUID="850bc9c2-fdcd-4a05-b7dc-1f04ea59f821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 01 06:37:53.038504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:53.029798 5040 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 01 06:37:53.038504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:53.029855 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" podUID="952eafb2-b8bc-457d-9bd6-4560f6176b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 01 06:37:53.969012 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:53.968971 5040 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 01 06:37:53.969223 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:53.969034 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" podUID="850bc9c2-fdcd-4a05-b7dc-1f04ea59f821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 01 06:37:54.043504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:54.040354 5040 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 01 06:37:54.043504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:54.040412 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" podUID="952eafb2-b8bc-457d-9bd6-4560f6176b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 01 06:37:55.031239 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:55.031200 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" Nov 01 06:37:55.032519 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:55.032314 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" Nov 01 06:37:55.033385 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:55.033345 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" Nov 01 06:37:55.096191 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:55.096154 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" Nov 01 06:37:55.097096 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:55.097068 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" Nov 01 06:37:55.098042 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:55.098010 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" Nov 01 06:37:56.948250 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:56.948216 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:37:57.109971 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.109929 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9577376-97a1-4915-b69e-26e60c29fce4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9577376-97a1-4915-b69e-26e60c29fce4-workload-certs\") pod \"d9577376-97a1-4915-b69e-26e60c29fce4\" (UID: \"d9577376-97a1-4915-b69e-26e60c29fce4\") " Nov 01 06:37:57.110135 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.110053 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9577376-97a1-4915-b69e-26e60c29fce4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9577376-97a1-4915-b69e-26e60c29fce4-istio-envoy\") pod \"d9577376-97a1-4915-b69e-26e60c29fce4\" (UID: \"d9577376-97a1-4915-b69e-26e60c29fce4\") " Nov 01 06:37:57.110135 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.110074 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9577376-97a1-4915-b69e-26e60c29fce4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9577376-97a1-4915-b69e-26e60c29fce4-istio-data\") pod \"d9577376-97a1-4915-b69e-26e60c29fce4\" (UID: \"d9577376-97a1-4915-b69e-26e60c29fce4\") " Nov 01 06:37:57.110135 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.110094 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9577376-97a1-4915-b69e-26e60c29fce4-istio-token\" (UniqueName: \"kubernetes.io/projected/d9577376-97a1-4915-b69e-26e60c29fce4-istio-token\") pod \"d9577376-97a1-4915-b69e-26e60c29fce4\" (UID: \"d9577376-97a1-4915-b69e-26e60c29fce4\") " Nov 01 06:37:57.110135 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.110136 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9577376-97a1-4915-b69e-26e60c29fce4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9577376-97a1-4915-b69e-26e60c29fce4-istio-podinfo\") pod \"d9577376-97a1-4915-b69e-26e60c29fce4\" (UID: \"d9577376-97a1-4915-b69e-26e60c29fce4\") " Nov 01 06:37:57.110420 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.110208 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9577376-97a1-4915-b69e-26e60c29fce4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9577376-97a1-4915-b69e-26e60c29fce4-workload-socket\") pod \"d9577376-97a1-4915-b69e-26e60c29fce4\" (UID: \"d9577376-97a1-4915-b69e-26e60c29fce4\") " Nov 01 06:37:57.110420 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.110229 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9577376-97a1-4915-b69e-26e60c29fce4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9577376-97a1-4915-b69e-26e60c29fce4-istiod-ca-cert\") pod \"d9577376-97a1-4915-b69e-26e60c29fce4\" (UID: \"d9577376-97a1-4915-b69e-26e60c29fce4\") " Nov 01 06:37:57.110420 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.110247 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9577376-97a1-4915-b69e-26e60c29fce4-workload-certs" pod "d9577376-97a1-4915-b69e-26e60c29fce4" (UID: "d9577376-97a1-4915-b69e-26e60c29fce4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:37:57.110420 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.110266 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9577376-97a1-4915-b69e-26e60c29fce4-kube-api-access-j9vnb\" (UniqueName: \"kubernetes.io/projected/d9577376-97a1-4915-b69e-26e60c29fce4-kube-api-access-j9vnb\") pod \"d9577376-97a1-4915-b69e-26e60c29fce4\" (UID: \"d9577376-97a1-4915-b69e-26e60c29fce4\") " Nov 01 06:37:57.110420 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.110306 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9577376-97a1-4915-b69e-26e60c29fce4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9577376-97a1-4915-b69e-26e60c29fce4-credential-socket\") pod \"d9577376-97a1-4915-b69e-26e60c29fce4\" (UID: \"d9577376-97a1-4915-b69e-26e60c29fce4\") " Nov 01 06:37:57.110747 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.110448 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9577376-97a1-4915-b69e-26e60c29fce4-istio-data" pod "d9577376-97a1-4915-b69e-26e60c29fce4" (UID: "d9577376-97a1-4915-b69e-26e60c29fce4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:37:57.110747 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.110553 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9577376-97a1-4915-b69e-26e60c29fce4-workload-socket" pod "d9577376-97a1-4915-b69e-26e60c29fce4" (UID: "d9577376-97a1-4915-b69e-26e60c29fce4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:37:57.110747 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.110723 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9577376-97a1-4915-b69e-26e60c29fce4-credential-socket" pod "d9577376-97a1-4915-b69e-26e60c29fce4" (UID: "d9577376-97a1-4915-b69e-26e60c29fce4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:37:57.110940 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.110741 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9577376-97a1-4915-b69e-26e60c29fce4-istiod-ca-cert" pod "d9577376-97a1-4915-b69e-26e60c29fce4" (UID: "d9577376-97a1-4915-b69e-26e60c29fce4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:37:57.110940 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.110927 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9577376-97a1-4915-b69e-26e60c29fce4-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:37:57.111049 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.110948 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9577376-97a1-4915-b69e-26e60c29fce4-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:37:57.111049 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.110963 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9577376-97a1-4915-b69e-26e60c29fce4-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:37:57.111049 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.110978 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9577376-97a1-4915-b69e-26e60c29fce4-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:37:57.111049 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.110993 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9577376-97a1-4915-b69e-26e60c29fce4-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:37:57.112565 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.112534 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9577376-97a1-4915-b69e-26e60c29fce4-istio-token" pod "d9577376-97a1-4915-b69e-26e60c29fce4" (UID: "d9577376-97a1-4915-b69e-26e60c29fce4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:37:57.112565 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.112546 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9577376-97a1-4915-b69e-26e60c29fce4-istio-podinfo" pod "d9577376-97a1-4915-b69e-26e60c29fce4" (UID: "d9577376-97a1-4915-b69e-26e60c29fce4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:37:57.112712 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.112569 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9577376-97a1-4915-b69e-26e60c29fce4-istio-envoy" pod "d9577376-97a1-4915-b69e-26e60c29fce4" (UID: "d9577376-97a1-4915-b69e-26e60c29fce4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:37:57.112712 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.112658 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9577376-97a1-4915-b69e-26e60c29fce4-kube-api-access-j9vnb" pod "d9577376-97a1-4915-b69e-26e60c29fce4" (UID: "d9577376-97a1-4915-b69e-26e60c29fce4"). InnerVolumeSpecName "kube-api-access-j9vnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:37:57.211720 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.211676 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9577376-97a1-4915-b69e-26e60c29fce4-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:37:57.211720 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.211717 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9577376-97a1-4915-b69e-26e60c29fce4-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:37:57.211720 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.211728 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9577376-97a1-4915-b69e-26e60c29fce4-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:37:57.211973 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.211738 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9vnb\" (UniqueName: \"kubernetes.io/projected/d9577376-97a1-4915-b69e-26e60c29fce4-kube-api-access-j9vnb\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:37:57.624744 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.624710 5040 generic.go:356] "Generic (PLEG): container finished" podID="d9577376-97a1-4915-b69e-26e60c29fce4" containerID="c1beeaff29df2cac39f16cd34c4fa153761bee8150cb413aa013d910af704a19" exitCode=0 Nov 01 06:37:57.624945 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.624773 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" event={"ID":"d9577376-97a1-4915-b69e-26e60c29fce4","Type":"ContainerDied","Data":"c1beeaff29df2cac39f16cd34c4fa153761bee8150cb413aa013d910af704a19"} Nov 01 06:37:57.624945 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.624828 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" event={"ID":"d9577376-97a1-4915-b69e-26e60c29fce4","Type":"ContainerDied","Data":"5b4dd48d84d752b362f13b97687af08cee0adf64c43d9b80dcb70122b78915e5"} Nov 01 06:37:57.624945 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.624836 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82" Nov 01 06:37:57.624945 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.624850 5040 scope.go:122] "RemoveContainer" containerID="c1beeaff29df2cac39f16cd34c4fa153761bee8150cb413aa013d910af704a19" Nov 01 06:37:57.635102 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.635086 5040 scope.go:122] "RemoveContainer" containerID="3dfe98b97d63aa55aa6310dee9141b058482d425f4b7115638bbf0e0bd5ea214" Nov 01 06:37:57.643899 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.643881 5040 scope.go:122] "RemoveContainer" containerID="c1beeaff29df2cac39f16cd34c4fa153761bee8150cb413aa013d910af704a19" Nov 01 06:37:57.644143 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:37:57.644127 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1beeaff29df2cac39f16cd34c4fa153761bee8150cb413aa013d910af704a19\": container with ID starting with c1beeaff29df2cac39f16cd34c4fa153761bee8150cb413aa013d910af704a19 not found: ID does not exist" containerID="c1beeaff29df2cac39f16cd34c4fa153761bee8150cb413aa013d910af704a19" Nov 01 06:37:57.644188 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.644152 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1beeaff29df2cac39f16cd34c4fa153761bee8150cb413aa013d910af704a19"} err="failed to get container status \"c1beeaff29df2cac39f16cd34c4fa153761bee8150cb413aa013d910af704a19\": rpc error: code = NotFound desc = could not find container \"c1beeaff29df2cac39f16cd34c4fa153761bee8150cb413aa013d910af704a19\": container with ID starting with c1beeaff29df2cac39f16cd34c4fa153761bee8150cb413aa013d910af704a19 not found: ID does not exist" Nov 01 06:37:57.644188 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.644170 5040 scope.go:122] "RemoveContainer" containerID="3dfe98b97d63aa55aa6310dee9141b058482d425f4b7115638bbf0e0bd5ea214" Nov 01 06:37:57.644401 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:37:57.644384 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dfe98b97d63aa55aa6310dee9141b058482d425f4b7115638bbf0e0bd5ea214\": container with ID starting with 3dfe98b97d63aa55aa6310dee9141b058482d425f4b7115638bbf0e0bd5ea214 not found: ID does not exist" containerID="3dfe98b97d63aa55aa6310dee9141b058482d425f4b7115638bbf0e0bd5ea214" Nov 01 06:37:57.644452 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.644405 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dfe98b97d63aa55aa6310dee9141b058482d425f4b7115638bbf0e0bd5ea214"} err="failed to get container status \"3dfe98b97d63aa55aa6310dee9141b058482d425f4b7115638bbf0e0bd5ea214\": rpc error: code = NotFound desc = could not find container \"3dfe98b97d63aa55aa6310dee9141b058482d425f4b7115638bbf0e0bd5ea214\": container with ID starting with 3dfe98b97d63aa55aa6310dee9141b058482d425f4b7115638bbf0e0bd5ea214 not found: ID does not exist" Nov 01 06:37:57.792975 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.792890 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82"] Nov 01 06:37:57.844492 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:57.844424 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-3680c5-openshift-default-5bdb59b98f-ttp82"] Nov 01 06:37:59.515269 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:37:59.515236 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9577376-97a1-4915-b69e-26e60c29fce4" path="/var/lib/kubelet/pods/d9577376-97a1-4915-b69e-26e60c29fce4/volumes" Nov 01 06:38:02.783834 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:02.783797 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc"] Nov 01 06:38:02.784580 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:02.784558 5040 cpu_manager.go:414] "containerMap: removing container" podUID="d9577376-97a1-4915-b69e-26e60c29fce4" containerName="istio-proxy" Nov 01 06:38:02.784654 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:02.784583 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9577376-97a1-4915-b69e-26e60c29fce4" containerName="istio-proxy" Nov 01 06:38:02.784654 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:02.784606 5040 cpu_manager.go:414] "containerMap: removing container" podUID="d9577376-97a1-4915-b69e-26e60c29fce4" containerName="istio-proxy" Nov 01 06:38:02.784654 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:02.784613 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9577376-97a1-4915-b69e-26e60c29fce4" containerName="istio-proxy" Nov 01 06:38:02.784826 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:02.784811 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9577376-97a1-4915-b69e-26e60c29fce4" containerName="istio-proxy" Nov 01 06:38:02.784860 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:02.784833 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9577376-97a1-4915-b69e-26e60c29fce4" containerName="istio-proxy" Nov 01 06:38:02.790463 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:02.790438 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" Nov 01 06:38:02.875786 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:02.875750 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc"] Nov 01 06:38:02.881201 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:02.881153 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-istiod-ca-cert\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-w44jc\" (UID: \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" Nov 01 06:38:02.881375 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:02.881257 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n26vp\" (UniqueName: \"kubernetes.io/projected/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-kube-api-access-n26vp\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-w44jc\" (UID: \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" Nov 01 06:38:02.881375 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:02.881322 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-workload-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-w44jc\" (UID: \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" Nov 01 06:38:02.881375 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:02.881355 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-credential-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-w44jc\" (UID: \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" Nov 01 06:38:02.881616 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:02.881380 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-workload-certs\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-w44jc\" (UID: \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" Nov 01 06:38:02.881616 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:02.881441 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-istio-token\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-w44jc\" (UID: \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" Nov 01 06:38:02.881616 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:02.881502 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-istio-data\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-w44jc\" (UID: \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" Nov 01 06:38:02.881616 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:02.881550 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-istio-envoy\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-w44jc\" (UID: \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" Nov 01 06:38:02.881616 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:02.881611 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-istio-podinfo\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-w44jc\" (UID: \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" Nov 01 06:38:02.984506 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:02.983611 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-workload-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-w44jc\" (UID: \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" Nov 01 06:38:02.984506 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:02.984075 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-workload-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-w44jc\" (UID: \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" Nov 01 06:38:02.984506 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:02.984145 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-credential-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-w44jc\" (UID: \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" Nov 01 06:38:02.984506 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:02.984173 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-workload-certs\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-w44jc\" (UID: \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" Nov 01 06:38:02.984506 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:02.984245 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-istio-token\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-w44jc\" (UID: \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" Nov 01 06:38:02.984506 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:02.984291 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-istio-data\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-w44jc\" (UID: \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" Nov 01 06:38:02.984506 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:02.984338 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-istio-envoy\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-w44jc\" (UID: \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" Nov 01 06:38:02.984506 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:02.984400 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-istio-podinfo\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-w44jc\" (UID: \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" Nov 01 06:38:02.984506 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:02.984455 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-istiod-ca-cert\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-w44jc\" (UID: \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" Nov 01 06:38:02.985750 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:02.985244 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n26vp\" (UniqueName: \"kubernetes.io/projected/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-kube-api-access-n26vp\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-w44jc\" (UID: \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" Nov 01 06:38:02.985881 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:02.985833 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-istio-data\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-w44jc\" (UID: \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" Nov 01 06:38:02.986100 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:02.986080 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-credential-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-w44jc\" (UID: \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" Nov 01 06:38:03.003700 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:02.986465 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-workload-certs\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-w44jc\" (UID: \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" Nov 01 06:38:03.003700 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:02.997802 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-istiod-ca-cert\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-w44jc\" (UID: \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" Nov 01 06:38:03.004205 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:03.004172 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-istio-envoy\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-w44jc\" (UID: \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" Nov 01 06:38:03.013813 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:03.013784 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-istio-podinfo\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-w44jc\" (UID: \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" Nov 01 06:38:03.052113 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:03.052022 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-istio-token\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-w44jc\" (UID: \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" Nov 01 06:38:03.056221 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:03.056189 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n26vp\" (UniqueName: \"kubernetes.io/projected/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-kube-api-access-n26vp\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-w44jc\" (UID: \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" Nov 01 06:38:03.111358 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:03.109747 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" Nov 01 06:38:03.587256 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:03.587221 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc"] Nov 01 06:38:03.587581 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:38:03.587556 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72fe0dc9_1d5e_4be1_a48c_51af8c3e578c.slice/crio-175ca02a83930a752d406881a83e5e6033c65a4bf4b5ccbe9718a09111e0f1f0 WatchSource:0}: Error finding container 175ca02a83930a752d406881a83e5e6033c65a4bf4b5ccbe9718a09111e0f1f0: Status 404 returned error can't find the container with id 175ca02a83930a752d406881a83e5e6033c65a4bf4b5ccbe9718a09111e0f1f0 Nov 01 06:38:03.590950 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:03.590912 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:38:03.591069 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:03.590994 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:38:03.591069 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:03.591038 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:38:03.675433 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:03.675397 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" event={"ID":"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c","Type":"ContainerStarted","Data":"175ca02a83930a752d406881a83e5e6033c65a4bf4b5ccbe9718a09111e0f1f0"} Nov 01 06:38:03.803263 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:03.803209 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" podStartSLOduration=1.8031951240000001 podStartE2EDuration="1.803195124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:38:03.800178418 +0000 UTC m=+3162.455295588" watchObservedRunningTime="2026-11-01 06:38:03.803195124 +0000 UTC m=+3162.458312290" Nov 01 06:38:04.111326 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:04.111276 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" Nov 01 06:38:04.112702 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:04.112679 5040 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 01 06:38:04.112816 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:04.112727 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" podUID="72fe0dc9-1d5e-4be1-a48c-51af8c3e578c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 01 06:38:04.260160 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:04.260125 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7134-380c9e"] Nov 01 06:38:04.264009 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:04.263992 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7134-380c9e" Nov 01 06:38:04.302474 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:04.302188 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pldc5\" (UniqueName: \"kubernetes.io/projected/0112ca41-862a-48ef-9ea5-775477c13bde-kube-api-access-pldc5\") pod \"test-curl-7134-380c9e\" (UID: \"0112ca41-862a-48ef-9ea5-775477c13bde\") " pod="opendatahub/test-curl-7134-380c9e" Nov 01 06:38:04.324566 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:04.323920 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hgn5l\"" Nov 01 06:38:04.328493 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:04.328453 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7134-380c9e"] Nov 01 06:38:04.418563 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:04.418403 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pldc5\" (UniqueName: \"kubernetes.io/projected/0112ca41-862a-48ef-9ea5-775477c13bde-kube-api-access-pldc5\") pod \"test-curl-7134-380c9e\" (UID: \"0112ca41-862a-48ef-9ea5-775477c13bde\") " pod="opendatahub/test-curl-7134-380c9e" Nov 01 06:38:04.480104 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:04.480073 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pldc5\" (UniqueName: \"kubernetes.io/projected/0112ca41-862a-48ef-9ea5-775477c13bde-kube-api-access-pldc5\") pod \"test-curl-7134-380c9e\" (UID: \"0112ca41-862a-48ef-9ea5-775477c13bde\") " pod="opendatahub/test-curl-7134-380c9e" Nov 01 06:38:04.576988 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:04.576956 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7134-380c9e" Nov 01 06:38:04.684167 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:04.684074 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" event={"ID":"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c","Type":"ContainerStarted","Data":"ff39e9bd0769289400ca9eab27647ddc6af1840776b133a1bac7f5ee88a92bbe"} Nov 01 06:38:05.021673 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:05.021642 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7134-380c9e"] Nov 01 06:38:05.023660 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:38:05.023631 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0112ca41_862a_48ef_9ea5_775477c13bde.slice/crio-772a97a8a426b1fbef2abb928e0a94b042493c455d80a1bf42e61d8a40688aad WatchSource:0}: Error finding container 772a97a8a426b1fbef2abb928e0a94b042493c455d80a1bf42e61d8a40688aad: Status 404 returned error can't find the container with id 772a97a8a426b1fbef2abb928e0a94b042493c455d80a1bf42e61d8a40688aad Nov 01 06:38:05.120549 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:05.119746 5040 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 01 06:38:05.120549 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:05.119806 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" podUID="72fe0dc9-1d5e-4be1-a48c-51af8c3e578c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 01 06:38:05.691421 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:05.691375 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7134-380c9e" event={"ID":"0112ca41-862a-48ef-9ea5-775477c13bde","Type":"ContainerStarted","Data":"f2a5a109316e77b11ed96cd0a435a9ee562d7fb510fa6ac841be15b6eeafe02c"} Nov 01 06:38:05.691421 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:05.691421 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7134-380c9e" event={"ID":"0112ca41-862a-48ef-9ea5-775477c13bde","Type":"ContainerStarted","Data":"772a97a8a426b1fbef2abb928e0a94b042493c455d80a1bf42e61d8a40688aad"} Nov 01 06:38:05.819782 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:05.819724 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7134-380c9e" podStartSLOduration=1.819709401 podStartE2EDuration="1.819709401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:38:05.818666431 +0000 UTC m=+3164.473783600" watchObservedRunningTime="2026-11-01 06:38:05.819709401 +0000 UTC m=+3164.474826567" Nov 01 06:38:06.113946 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:06.113893 5040 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 01 06:38:06.114563 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:06.114528 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" podUID="72fe0dc9-1d5e-4be1-a48c-51af8c3e578c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 01 06:38:07.118505 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:07.112901 5040 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 01 06:38:07.118505 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:07.112967 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" podUID="72fe0dc9-1d5e-4be1-a48c-51af8c3e578c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 01 06:38:07.287767 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:07.287731 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7134-380c9e"] Nov 01 06:38:07.291978 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:07.291947 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7134-380c9e"] Nov 01 06:38:07.744085 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:07.744044 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7134-380c9e" podUID="0112ca41-862a-48ef-9ea5-775477c13bde" containerName="test-curl-7134-380c9e" containerID="cri-o://f2a5a109316e77b11ed96cd0a435a9ee562d7fb510fa6ac841be15b6eeafe02c" gracePeriod=2 Nov 01 06:38:08.114503 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:08.113918 5040 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 01 06:38:08.114503 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:08.113979 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" podUID="72fe0dc9-1d5e-4be1-a48c-51af8c3e578c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 01 06:38:09.114503 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:09.113862 5040 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 01 06:38:09.114503 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:09.113925 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" podUID="72fe0dc9-1d5e-4be1-a48c-51af8c3e578c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 01 06:38:09.978094 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:38:09.978052 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:38:10.012353 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:10.012294 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7134-380c9e" Nov 01 06:38:10.089865 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:10.089826 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0112ca41-862a-48ef-9ea5-775477c13bde-kube-api-access-pldc5\" (UniqueName: \"kubernetes.io/projected/0112ca41-862a-48ef-9ea5-775477c13bde-kube-api-access-pldc5\") pod \"0112ca41-862a-48ef-9ea5-775477c13bde\" (UID: \"0112ca41-862a-48ef-9ea5-775477c13bde\") " Nov 01 06:38:10.091933 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:10.091907 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0112ca41-862a-48ef-9ea5-775477c13bde-kube-api-access-pldc5" pod "0112ca41-862a-48ef-9ea5-775477c13bde" (UID: "0112ca41-862a-48ef-9ea5-775477c13bde"). InnerVolumeSpecName "kube-api-access-pldc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:38:10.130573 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:10.130527 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" Nov 01 06:38:10.131065 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:10.130971 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" Nov 01 06:38:10.132185 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:10.132165 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" Nov 01 06:38:10.193137 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:10.192426 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pldc5\" (UniqueName: \"kubernetes.io/projected/0112ca41-862a-48ef-9ea5-775477c13bde-kube-api-access-pldc5\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:38:10.765833 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:10.765796 5040 generic.go:356] "Generic (PLEG): container finished" podID="0112ca41-862a-48ef-9ea5-775477c13bde" containerID="f2a5a109316e77b11ed96cd0a435a9ee562d7fb510fa6ac841be15b6eeafe02c" exitCode=137 Nov 01 06:38:10.766025 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:10.765855 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7134-380c9e" Nov 01 06:38:10.766025 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:10.765875 5040 scope.go:122] "RemoveContainer" containerID="f2a5a109316e77b11ed96cd0a435a9ee562d7fb510fa6ac841be15b6eeafe02c" Nov 01 06:38:10.777303 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:10.777285 5040 scope.go:122] "RemoveContainer" containerID="f2a5a109316e77b11ed96cd0a435a9ee562d7fb510fa6ac841be15b6eeafe02c" Nov 01 06:38:10.777613 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:38:10.777590 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2a5a109316e77b11ed96cd0a435a9ee562d7fb510fa6ac841be15b6eeafe02c\": container with ID starting with f2a5a109316e77b11ed96cd0a435a9ee562d7fb510fa6ac841be15b6eeafe02c not found: ID does not exist" containerID="f2a5a109316e77b11ed96cd0a435a9ee562d7fb510fa6ac841be15b6eeafe02c" Nov 01 06:38:10.777692 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:10.777624 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2a5a109316e77b11ed96cd0a435a9ee562d7fb510fa6ac841be15b6eeafe02c"} err="failed to get container status \"f2a5a109316e77b11ed96cd0a435a9ee562d7fb510fa6ac841be15b6eeafe02c\": rpc error: code = NotFound desc = could not find container \"f2a5a109316e77b11ed96cd0a435a9ee562d7fb510fa6ac841be15b6eeafe02c\": container with ID starting with f2a5a109316e77b11ed96cd0a435a9ee562d7fb510fa6ac841be15b6eeafe02c not found: ID does not exist" Nov 01 06:38:10.854296 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:38:10.853884 5040 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7134-380c9e\" is forbidden: User \"system:node:ip-10-0-143-195.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-195.us-west-1.compute.internal' and this object" podUID="0112ca41-862a-48ef-9ea5-775477c13bde" pod="opendatahub/test-curl-7134-380c9e" Nov 01 06:38:11.515770 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:11.515732 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0112ca41-862a-48ef-9ea5-775477c13bde" path="/var/lib/kubelet/pods/0112ca41-862a-48ef-9ea5-775477c13bde/volumes" Nov 01 06:38:11.564308 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:38:11.564274 5040 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7134-380c9e\" is forbidden: User \"system:node:ip-10-0-143-195.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-195.us-west-1.compute.internal' and this object" podUID="0112ca41-862a-48ef-9ea5-775477c13bde" pod="opendatahub/test-curl-7134-380c9e" Nov 01 06:38:12.787197 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:12.787168 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv"] Nov 01 06:38:23.437447 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:23.437411 5040 scope.go:122] "RemoveContainer" containerID="9ac2d9b451bf6bb4075fbe6ee9422d3bcd167b69c31a47e185095754297d74e8" Nov 01 06:38:27.809426 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:27.809364 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv" podUID="72ab782a-6494-430b-a5eb-17f4b4f9ddd3" containerName="main" containerID="cri-o://390d5f45fdf9571d0d91af0185ecbd9f98c5914de8a5af1f15c4cb136f732c3f" gracePeriod=45 Nov 01 06:38:28.074626 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:28.074566 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv" Nov 01 06:38:28.098306 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:28.098271 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-tls-certs\" (UniqueName: \"kubernetes.io/secret/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-tls-certs\") pod \"72ab782a-6494-430b-a5eb-17f4b4f9ddd3\" (UID: \"72ab782a-6494-430b-a5eb-17f4b4f9ddd3\") " Nov 01 06:38:28.098469 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:28.098333 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-home\" (UniqueName: \"kubernetes.io/empty-dir/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-home\") pod \"72ab782a-6494-430b-a5eb-17f4b4f9ddd3\" (UID: \"72ab782a-6494-430b-a5eb-17f4b4f9ddd3\") " Nov 01 06:38:28.098529 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:28.098494 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-dshm\") pod \"72ab782a-6494-430b-a5eb-17f4b4f9ddd3\" (UID: \"72ab782a-6494-430b-a5eb-17f4b4f9ddd3\") " Nov 01 06:38:28.098572 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:28.098548 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-kube-api-access-ws8j9\" (UniqueName: \"kubernetes.io/projected/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-kube-api-access-ws8j9\") pod \"72ab782a-6494-430b-a5eb-17f4b4f9ddd3\" (UID: \"72ab782a-6494-430b-a5eb-17f4b4f9ddd3\") " Nov 01 06:38:28.098634 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:28.098614 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-tmp-dir\") pod \"72ab782a-6494-430b-a5eb-17f4b4f9ddd3\" (UID: \"72ab782a-6494-430b-a5eb-17f4b4f9ddd3\") " Nov 01 06:38:28.098695 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:28.098650 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-home" pod "72ab782a-6494-430b-a5eb-17f4b4f9ddd3" (UID: "72ab782a-6494-430b-a5eb-17f4b4f9ddd3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:38:28.098783 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:28.098757 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-model-cache\") pod \"72ab782a-6494-430b-a5eb-17f4b4f9ddd3\" (UID: \"72ab782a-6494-430b-a5eb-17f4b4f9ddd3\") " Nov 01 06:38:28.098898 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:28.098877 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-tmp-dir" pod "72ab782a-6494-430b-a5eb-17f4b4f9ddd3" (UID: "72ab782a-6494-430b-a5eb-17f4b4f9ddd3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:38:28.099030 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:28.099000 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-model-cache" pod "72ab782a-6494-430b-a5eb-17f4b4f9ddd3" (UID: "72ab782a-6494-430b-a5eb-17f4b4f9ddd3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:38:28.099203 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:28.099187 5040 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-tmp-dir\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:38:28.099281 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:28.099204 5040 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-model-cache\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:38:28.099281 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:28.099214 5040 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-home\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:38:28.100700 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:28.100669 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-dshm" pod "72ab782a-6494-430b-a5eb-17f4b4f9ddd3" (UID: "72ab782a-6494-430b-a5eb-17f4b4f9ddd3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:38:28.100791 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:28.100721 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-tls-certs" pod "72ab782a-6494-430b-a5eb-17f4b4f9ddd3" (UID: "72ab782a-6494-430b-a5eb-17f4b4f9ddd3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:38:28.100791 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:28.100753 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-kube-api-access-ws8j9" pod "72ab782a-6494-430b-a5eb-17f4b4f9ddd3" (UID: "72ab782a-6494-430b-a5eb-17f4b4f9ddd3"). InnerVolumeSpecName "kube-api-access-ws8j9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:38:28.200959 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:28.200921 5040 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-tls-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:38:28.200959 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:28.200961 5040 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-dshm\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:38:28.201218 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:28.200976 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws8j9\" (UniqueName: \"kubernetes.io/projected/72ab782a-6494-430b-a5eb-17f4b4f9ddd3-kube-api-access-ws8j9\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:38:28.882784 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:28.882742 5040 generic.go:356] "Generic (PLEG): container finished" podID="72ab782a-6494-430b-a5eb-17f4b4f9ddd3" containerID="390d5f45fdf9571d0d91af0185ecbd9f98c5914de8a5af1f15c4cb136f732c3f" exitCode=0 Nov 01 06:38:28.883231 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:28.882790 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv" Nov 01 06:38:28.883231 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:28.882823 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv" event={"ID":"72ab782a-6494-430b-a5eb-17f4b4f9ddd3","Type":"ContainerDied","Data":"390d5f45fdf9571d0d91af0185ecbd9f98c5914de8a5af1f15c4cb136f732c3f"} Nov 01 06:38:28.883231 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:28.882874 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv" event={"ID":"72ab782a-6494-430b-a5eb-17f4b4f9ddd3","Type":"ContainerDied","Data":"a6a25b0f3c63c12bd11dd135ab33dcabc0496dfa05ec3f97f813d6be57f36ea5"} Nov 01 06:38:28.883231 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:28.882892 5040 scope.go:122] "RemoveContainer" containerID="390d5f45fdf9571d0d91af0185ecbd9f98c5914de8a5af1f15c4cb136f732c3f" Nov 01 06:38:28.894021 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:28.893999 5040 scope.go:122] "RemoveContainer" containerID="390d5f45fdf9571d0d91af0185ecbd9f98c5914de8a5af1f15c4cb136f732c3f" Nov 01 06:38:28.894289 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:38:28.894272 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"390d5f45fdf9571d0d91af0185ecbd9f98c5914de8a5af1f15c4cb136f732c3f\": container with ID starting with 390d5f45fdf9571d0d91af0185ecbd9f98c5914de8a5af1f15c4cb136f732c3f not found: ID does not exist" containerID="390d5f45fdf9571d0d91af0185ecbd9f98c5914de8a5af1f15c4cb136f732c3f" Nov 01 06:38:28.894339 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:28.894296 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"390d5f45fdf9571d0d91af0185ecbd9f98c5914de8a5af1f15c4cb136f732c3f"} err="failed to get container status \"390d5f45fdf9571d0d91af0185ecbd9f98c5914de8a5af1f15c4cb136f732c3f\": rpc error: code = NotFound desc = could not find container \"390d5f45fdf9571d0d91af0185ecbd9f98c5914de8a5af1f15c4cb136f732c3f\": container with ID starting with 390d5f45fdf9571d0d91af0185ecbd9f98c5914de8a5af1f15c4cb136f732c3f not found: ID does not exist" Nov 01 06:38:29.050049 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:29.050009 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv"] Nov 01 06:38:29.100742 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:29.100705 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-c0662f25/e2e-lifecycle-model-c0662f25-kserve-756794857-fpcvv"] Nov 01 06:38:29.515031 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:29.514994 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72ab782a-6494-430b-a5eb-17f4b4f9ddd3" path="/var/lib/kubelet/pods/72ab782a-6494-430b-a5eb-17f4b4f9ddd3/volumes" Nov 01 06:38:39.974706 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:38:39.974624 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:38:47.679891 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:47.679856 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2320dfee-rvn6q"] Nov 01 06:38:47.680643 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:47.680622 5040 cpu_manager.go:414] "containerMap: removing container" podUID="0112ca41-862a-48ef-9ea5-775477c13bde" containerName="test-curl-7134-380c9e" Nov 01 06:38:47.680752 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:47.680644 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0112ca41-862a-48ef-9ea5-775477c13bde" containerName="test-curl-7134-380c9e" Nov 01 06:38:47.680752 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:47.680691 5040 cpu_manager.go:414] "containerMap: removing container" podUID="72ab782a-6494-430b-a5eb-17f4b4f9ddd3" containerName="main" Nov 01 06:38:47.680752 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:47.680700 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72ab782a-6494-430b-a5eb-17f4b4f9ddd3" containerName="main" Nov 01 06:38:47.680928 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:47.680833 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="0112ca41-862a-48ef-9ea5-775477c13bde" containerName="test-curl-7134-380c9e" Nov 01 06:38:47.680928 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:47.680850 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="72ab782a-6494-430b-a5eb-17f4b4f9ddd3" containerName="main" Nov 01 06:38:47.685743 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:47.685720 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2320dfee-rvn6q" Nov 01 06:38:47.739041 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:47.739010 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:38:47.739280 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:47.739259 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:38:47.739361 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:47.739299 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:38:47.739641 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:47.739605 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:38:47.741533 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:47.741509 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2320dfee-rvn6q"] Nov 01 06:38:47.822591 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:47.822554 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5db3853a-2372-4864-9b1c-baa9d60a3fbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2320dfee-rvn6q\" (UID: \"5db3853a-2372-4864-9b1c-baa9d60a3fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2320dfee-rvn6q" Nov 01 06:38:47.822794 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:47.822623 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5db3853a-2372-4864-9b1c-baa9d60a3fbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2320dfee-rvn6q\" (UID: \"5db3853a-2372-4864-9b1c-baa9d60a3fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2320dfee-rvn6q" Nov 01 06:38:47.822876 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:47.822786 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5bbp\" (UniqueName: \"kubernetes.io/projected/5db3853a-2372-4864-9b1c-baa9d60a3fbc-kube-api-access-d5bbp\") pod \"maas-api-revoke-sub-2320dfee-rvn6q\" (UID: \"5db3853a-2372-4864-9b1c-baa9d60a3fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2320dfee-rvn6q" Nov 01 06:38:47.924168 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:47.924132 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5db3853a-2372-4864-9b1c-baa9d60a3fbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2320dfee-rvn6q\" (UID: \"5db3853a-2372-4864-9b1c-baa9d60a3fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2320dfee-rvn6q" Nov 01 06:38:47.924358 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:47.924228 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5bbp\" (UniqueName: \"kubernetes.io/projected/5db3853a-2372-4864-9b1c-baa9d60a3fbc-kube-api-access-d5bbp\") pod \"maas-api-revoke-sub-2320dfee-rvn6q\" (UID: \"5db3853a-2372-4864-9b1c-baa9d60a3fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2320dfee-rvn6q" Nov 01 06:38:47.924358 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:47.924320 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5db3853a-2372-4864-9b1c-baa9d60a3fbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2320dfee-rvn6q\" (UID: \"5db3853a-2372-4864-9b1c-baa9d60a3fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2320dfee-rvn6q" Nov 01 06:38:47.924884 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:47.924864 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5db3853a-2372-4864-9b1c-baa9d60a3fbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2320dfee-rvn6q\" (UID: \"5db3853a-2372-4864-9b1c-baa9d60a3fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2320dfee-rvn6q" Nov 01 06:38:47.926644 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:47.926627 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5db3853a-2372-4864-9b1c-baa9d60a3fbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2320dfee-rvn6q\" (UID: \"5db3853a-2372-4864-9b1c-baa9d60a3fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2320dfee-rvn6q" Nov 01 06:38:47.985981 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:47.985904 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5bbp\" (UniqueName: \"kubernetes.io/projected/5db3853a-2372-4864-9b1c-baa9d60a3fbc-kube-api-access-d5bbp\") pod \"maas-api-revoke-sub-2320dfee-rvn6q\" (UID: \"5db3853a-2372-4864-9b1c-baa9d60a3fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2320dfee-rvn6q" Nov 01 06:38:47.999738 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:47.999710 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2320dfee-rvn6q" Nov 01 06:38:48.380653 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:48.380621 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2320dfee-rvn6q"] Nov 01 06:38:48.381304 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:38:48.381276 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5db3853a_2372_4864_9b1c_baa9d60a3fbc.slice/crio-67ed89a7032dd753763cfa59a9db37ff4da1e5f750d9e9a9a5ef4c0b649dca08 WatchSource:0}: Error finding container 67ed89a7032dd753763cfa59a9db37ff4da1e5f750d9e9a9a5ef4c0b649dca08: Status 404 returned error can't find the container with id 67ed89a7032dd753763cfa59a9db37ff4da1e5f750d9e9a9a5ef4c0b649dca08 Nov 01 06:38:49.015154 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:49.015113 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2320dfee-rvn6q" event={"ID":"5db3853a-2372-4864-9b1c-baa9d60a3fbc","Type":"ContainerStarted","Data":"0d1f0560b5595f08203d97477b116efe5761c5b227e85f07d7907e7b04d6b253"} Nov 01 06:38:49.015580 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:49.015180 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2320dfee-rvn6q" event={"ID":"5db3853a-2372-4864-9b1c-baa9d60a3fbc","Type":"ContainerStarted","Data":"67ed89a7032dd753763cfa59a9db37ff4da1e5f750d9e9a9a5ef4c0b649dca08"} Nov 01 06:38:49.132389 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:49.132331 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2320dfee-rvn6q" podStartSLOduration=2.132305954 podStartE2EDuration="2.132305954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:38:49.131132335 +0000 UTC m=+3207.786249501" watchObservedRunningTime="2026-11-01 06:38:49.132305954 +0000 UTC m=+3207.787423121" Nov 01 06:38:50.023349 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:50.023314 5040 generic.go:356] "Generic (PLEG): container finished" podID="5db3853a-2372-4864-9b1c-baa9d60a3fbc" containerID="0d1f0560b5595f08203d97477b116efe5761c5b227e85f07d7907e7b04d6b253" exitCode=0 Nov 01 06:38:50.023761 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:50.023395 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2320dfee-rvn6q" event={"ID":"5db3853a-2372-4864-9b1c-baa9d60a3fbc","Type":"ContainerDied","Data":"0d1f0560b5595f08203d97477b116efe5761c5b227e85f07d7907e7b04d6b253"} Nov 01 06:38:51.193946 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:51.193914 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2320dfee-rvn6q" Nov 01 06:38:51.279311 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:51.279271 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5db3853a-2372-4864-9b1c-baa9d60a3fbc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5db3853a-2372-4864-9b1c-baa9d60a3fbc-maas-api-openshift-ca\") pod \"5db3853a-2372-4864-9b1c-baa9d60a3fbc\" (UID: \"5db3853a-2372-4864-9b1c-baa9d60a3fbc\") " Nov 01 06:38:51.279608 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:51.279363 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5db3853a-2372-4864-9b1c-baa9d60a3fbc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5db3853a-2372-4864-9b1c-baa9d60a3fbc-maas-api-xks-ca\") pod \"5db3853a-2372-4864-9b1c-baa9d60a3fbc\" (UID: \"5db3853a-2372-4864-9b1c-baa9d60a3fbc\") " Nov 01 06:38:51.279608 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:51.279468 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5db3853a-2372-4864-9b1c-baa9d60a3fbc-kube-api-access-d5bbp\" (UniqueName: \"kubernetes.io/projected/5db3853a-2372-4864-9b1c-baa9d60a3fbc-kube-api-access-d5bbp\") pod \"5db3853a-2372-4864-9b1c-baa9d60a3fbc\" (UID: \"5db3853a-2372-4864-9b1c-baa9d60a3fbc\") " Nov 01 06:38:51.280676 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:51.280596 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5db3853a-2372-4864-9b1c-baa9d60a3fbc-maas-api-openshift-ca" pod "5db3853a-2372-4864-9b1c-baa9d60a3fbc" (UID: "5db3853a-2372-4864-9b1c-baa9d60a3fbc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:38:51.285258 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:51.282907 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5db3853a-2372-4864-9b1c-baa9d60a3fbc-maas-api-xks-ca" pod "5db3853a-2372-4864-9b1c-baa9d60a3fbc" (UID: "5db3853a-2372-4864-9b1c-baa9d60a3fbc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:38:51.285258 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:51.282950 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db3853a-2372-4864-9b1c-baa9d60a3fbc-kube-api-access-d5bbp" pod "5db3853a-2372-4864-9b1c-baa9d60a3fbc" (UID: "5db3853a-2372-4864-9b1c-baa9d60a3fbc"). InnerVolumeSpecName "kube-api-access-d5bbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:38:51.381574 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:51.381530 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5db3853a-2372-4864-9b1c-baa9d60a3fbc-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:38:51.381574 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:51.381571 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5db3853a-2372-4864-9b1c-baa9d60a3fbc-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:38:51.381785 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:51.381587 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5bbp\" (UniqueName: \"kubernetes.io/projected/5db3853a-2372-4864-9b1c-baa9d60a3fbc-kube-api-access-d5bbp\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:38:52.040720 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:52.040686 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2320dfee-rvn6q" Nov 01 06:38:52.040926 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:52.040720 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2320dfee-rvn6q" event={"ID":"5db3853a-2372-4864-9b1c-baa9d60a3fbc","Type":"ContainerDied","Data":"67ed89a7032dd753763cfa59a9db37ff4da1e5f750d9e9a9a5ef4c0b649dca08"} Nov 01 06:38:52.040926 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:38:52.040814 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67ed89a7032dd753763cfa59a9db37ff4da1e5f750d9e9a9a5ef4c0b649dca08" Nov 01 06:39:01.583570 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:01.583526 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5e240d4-z5jww"] Nov 01 06:39:01.584259 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:01.584243 5040 cpu_manager.go:414] "containerMap: removing container" podUID="5db3853a-2372-4864-9b1c-baa9d60a3fbc" containerName="revoke-keys" Nov 01 06:39:01.584307 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:01.584261 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5db3853a-2372-4864-9b1c-baa9d60a3fbc" containerName="revoke-keys" Nov 01 06:39:01.584442 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:01.584430 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="5db3853a-2372-4864-9b1c-baa9d60a3fbc" containerName="revoke-keys" Nov 01 06:39:01.588352 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:01.588331 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e240d4-z5jww" Nov 01 06:39:01.618299 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:01.618263 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn4jp\" (UniqueName: \"kubernetes.io/projected/e2b7621d-9ec9-47d9-a515-c9d62ca39799-kube-api-access-xn4jp\") pod \"maas-api-revoke-sub-c5e240d4-z5jww\" (UID: \"e2b7621d-9ec9-47d9-a515-c9d62ca39799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e240d4-z5jww" Nov 01 06:39:01.618458 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:01.618316 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2b7621d-9ec9-47d9-a515-c9d62ca39799-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5e240d4-z5jww\" (UID: \"e2b7621d-9ec9-47d9-a515-c9d62ca39799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e240d4-z5jww" Nov 01 06:39:01.618458 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:01.618439 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2b7621d-9ec9-47d9-a515-c9d62ca39799-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5e240d4-z5jww\" (UID: \"e2b7621d-9ec9-47d9-a515-c9d62ca39799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e240d4-z5jww" Nov 01 06:39:01.643856 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:01.643822 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5e240d4-z5jww"] Nov 01 06:39:01.690451 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:01.690422 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:39:01.691203 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:01.691171 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:39:01.691394 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:01.691206 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:39:01.691506 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:01.691300 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:39:01.719979 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:01.719942 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn4jp\" (UniqueName: \"kubernetes.io/projected/e2b7621d-9ec9-47d9-a515-c9d62ca39799-kube-api-access-xn4jp\") pod \"maas-api-revoke-sub-c5e240d4-z5jww\" (UID: \"e2b7621d-9ec9-47d9-a515-c9d62ca39799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e240d4-z5jww" Nov 01 06:39:01.720152 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:01.719990 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2b7621d-9ec9-47d9-a515-c9d62ca39799-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5e240d4-z5jww\" (UID: \"e2b7621d-9ec9-47d9-a515-c9d62ca39799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e240d4-z5jww" Nov 01 06:39:01.720152 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:01.720018 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2b7621d-9ec9-47d9-a515-c9d62ca39799-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5e240d4-z5jww\" (UID: \"e2b7621d-9ec9-47d9-a515-c9d62ca39799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e240d4-z5jww" Nov 01 06:39:01.720704 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:01.720683 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2b7621d-9ec9-47d9-a515-c9d62ca39799-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5e240d4-z5jww\" (UID: \"e2b7621d-9ec9-47d9-a515-c9d62ca39799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e240d4-z5jww" Nov 01 06:39:01.722389 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:01.722356 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2b7621d-9ec9-47d9-a515-c9d62ca39799-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5e240d4-z5jww\" (UID: \"e2b7621d-9ec9-47d9-a515-c9d62ca39799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e240d4-z5jww" Nov 01 06:39:01.781949 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:01.781911 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn4jp\" (UniqueName: \"kubernetes.io/projected/e2b7621d-9ec9-47d9-a515-c9d62ca39799-kube-api-access-xn4jp\") pod \"maas-api-revoke-sub-c5e240d4-z5jww\" (UID: \"e2b7621d-9ec9-47d9-a515-c9d62ca39799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e240d4-z5jww" Nov 01 06:39:01.901947 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:01.901860 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e240d4-z5jww" Nov 01 06:39:02.279917 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:02.279885 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5e240d4-z5jww"] Nov 01 06:39:02.281082 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:39:02.281056 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2b7621d_9ec9_47d9_a515_c9d62ca39799.slice/crio-97d52325d4d2ff5a6b01a9488ce838fec55c10140fd9eed891962358a05eee6a WatchSource:0}: Error finding container 97d52325d4d2ff5a6b01a9488ce838fec55c10140fd9eed891962358a05eee6a: Status 404 returned error can't find the container with id 97d52325d4d2ff5a6b01a9488ce838fec55c10140fd9eed891962358a05eee6a Nov 01 06:39:03.117248 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:03.117206 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e240d4-z5jww" event={"ID":"e2b7621d-9ec9-47d9-a515-c9d62ca39799","Type":"ContainerStarted","Data":"1f00849015fa332305dc38a67009a848fa048abe9090414d368d90b884c26e60"} Nov 01 06:39:03.117762 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:03.117276 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e240d4-z5jww" event={"ID":"e2b7621d-9ec9-47d9-a515-c9d62ca39799","Type":"ContainerStarted","Data":"97d52325d4d2ff5a6b01a9488ce838fec55c10140fd9eed891962358a05eee6a"} Nov 01 06:39:03.236887 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:03.236837 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e240d4-z5jww" podStartSLOduration=2.23682342 podStartE2EDuration="2.23682342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:39:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:39:03.234401098 +0000 UTC m=+3221.889518268" watchObservedRunningTime="2026-11-01 06:39:03.23682342 +0000 UTC m=+3221.891940587" Nov 01 06:39:04.124700 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:04.124664 5040 generic.go:356] "Generic (PLEG): container finished" podID="e2b7621d-9ec9-47d9-a515-c9d62ca39799" containerID="1f00849015fa332305dc38a67009a848fa048abe9090414d368d90b884c26e60" exitCode=0 Nov 01 06:39:04.125134 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:04.124757 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e240d4-z5jww" event={"ID":"e2b7621d-9ec9-47d9-a515-c9d62ca39799","Type":"ContainerDied","Data":"1f00849015fa332305dc38a67009a848fa048abe9090414d368d90b884c26e60"} Nov 01 06:39:05.278816 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:05.278777 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e240d4-z5jww" Nov 01 06:39:05.362701 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:05.362665 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2b7621d-9ec9-47d9-a515-c9d62ca39799-kube-api-access-xn4jp\" (UniqueName: \"kubernetes.io/projected/e2b7621d-9ec9-47d9-a515-c9d62ca39799-kube-api-access-xn4jp\") pod \"e2b7621d-9ec9-47d9-a515-c9d62ca39799\" (UID: \"e2b7621d-9ec9-47d9-a515-c9d62ca39799\") " Nov 01 06:39:05.362895 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:05.362755 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2b7621d-9ec9-47d9-a515-c9d62ca39799-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2b7621d-9ec9-47d9-a515-c9d62ca39799-maas-api-xks-ca\") pod \"e2b7621d-9ec9-47d9-a515-c9d62ca39799\" (UID: \"e2b7621d-9ec9-47d9-a515-c9d62ca39799\") " Nov 01 06:39:05.362895 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:05.362878 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2b7621d-9ec9-47d9-a515-c9d62ca39799-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2b7621d-9ec9-47d9-a515-c9d62ca39799-maas-api-openshift-ca\") pod \"e2b7621d-9ec9-47d9-a515-c9d62ca39799\" (UID: \"e2b7621d-9ec9-47d9-a515-c9d62ca39799\") " Nov 01 06:39:05.363281 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:05.363245 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2b7621d-9ec9-47d9-a515-c9d62ca39799-maas-api-openshift-ca" pod "e2b7621d-9ec9-47d9-a515-c9d62ca39799" (UID: "e2b7621d-9ec9-47d9-a515-c9d62ca39799"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:39:05.363429 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:05.363383 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2b7621d-9ec9-47d9-a515-c9d62ca39799-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:39:05.364827 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:05.364795 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2b7621d-9ec9-47d9-a515-c9d62ca39799-kube-api-access-xn4jp" pod "e2b7621d-9ec9-47d9-a515-c9d62ca39799" (UID: "e2b7621d-9ec9-47d9-a515-c9d62ca39799"). InnerVolumeSpecName "kube-api-access-xn4jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:39:05.364944 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:05.364883 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2b7621d-9ec9-47d9-a515-c9d62ca39799-maas-api-xks-ca" pod "e2b7621d-9ec9-47d9-a515-c9d62ca39799" (UID: "e2b7621d-9ec9-47d9-a515-c9d62ca39799"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:39:05.464178 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:05.464095 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn4jp\" (UniqueName: \"kubernetes.io/projected/e2b7621d-9ec9-47d9-a515-c9d62ca39799-kube-api-access-xn4jp\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:39:05.464178 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:05.464128 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2b7621d-9ec9-47d9-a515-c9d62ca39799-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:39:06.139035 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:06.139006 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e240d4-z5jww" Nov 01 06:39:06.139225 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:06.139007 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e240d4-z5jww" event={"ID":"e2b7621d-9ec9-47d9-a515-c9d62ca39799","Type":"ContainerDied","Data":"97d52325d4d2ff5a6b01a9488ce838fec55c10140fd9eed891962358a05eee6a"} Nov 01 06:39:06.139225 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:06.139139 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97d52325d4d2ff5a6b01a9488ce838fec55c10140fd9eed891962358a05eee6a" Nov 01 06:39:10.022822 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:39:10.022782 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:39:11.204990 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:11.204946 5040 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp" podUID="8acef8ba-1ac8-4cfb-b853-bb9a63afe24e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.205:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:39:11.267526 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:11.254629 5040 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" podUID="75afe340-0b9c-4b4c-89ab-20d761e6b4f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:39:11.301503 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:11.281586 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" podUID="5daa03ca-824b-4799-a11b-4b3af1512c38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.123:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:39:11.549505 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:11.547469 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl" podUID="21f54836-d1ed-4f87-8f26-7c2d939b2590" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 06:39:11.549505 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:11.548067 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" podUID="75afe340-0b9c-4b4c-89ab-20d761e6b4f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:39:23.471033 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:23.470998 5040 scope.go:122] "RemoveContainer" containerID="ddc9231b454a02f9676e3a88bd37775e562929fc2f60bcd0b4ad1c8b81810f95" Nov 01 06:39:31.051748 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:31.051709 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd99b3db-652hl"] Nov 01 06:39:31.101701 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:31.101662 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd99b3db-652hl"] Nov 01 06:39:31.515598 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:31.515565 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df21123c-bfab-43d6-ae4d-a5a44c762b6f" path="/var/lib/kubelet/pods/df21123c-bfab-43d6-ae4d-a5a44c762b6f/volumes" Nov 01 06:39:38.721891 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:38.721854 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-m6c5j"] Nov 01 06:39:38.723369 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:38.723341 5040 cpu_manager.go:414] "containerMap: removing container" podUID="e2b7621d-9ec9-47d9-a515-c9d62ca39799" containerName="revoke-keys" Nov 01 06:39:38.723555 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:38.723538 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2b7621d-9ec9-47d9-a515-c9d62ca39799" containerName="revoke-keys" Nov 01 06:39:38.723843 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:38.723826 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2b7621d-9ec9-47d9-a515-c9d62ca39799" containerName="revoke-keys" Nov 01 06:39:38.729468 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:38.729444 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-m6c5j" Nov 01 06:39:38.806504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:38.784849 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-m6c5j"] Nov 01 06:39:38.824524 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:38.806891 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:39:38.824524 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:38.806987 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:39:38.824524 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:38.807016 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:39:38.824524 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:38.807648 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:39:38.824524 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:38.808337 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e5ef051-999a-4ab8-bf50-568c17f68ce1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c0662f25-m6c5j\" (UID: \"5e5ef051-999a-4ab8-bf50-568c17f68ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-m6c5j" Nov 01 06:39:38.824524 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:38.815749 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e5ef051-999a-4ab8-bf50-568c17f68ce1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c0662f25-m6c5j\" (UID: \"5e5ef051-999a-4ab8-bf50-568c17f68ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-m6c5j" Nov 01 06:39:38.824524 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:38.815913 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp24x\" (UniqueName: \"kubernetes.io/projected/5e5ef051-999a-4ab8-bf50-568c17f68ce1-kube-api-access-rp24x\") pod \"maas-api-revoke-keys-e2e-mt-c0662f25-m6c5j\" (UID: \"5e5ef051-999a-4ab8-bf50-568c17f68ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-m6c5j" Nov 01 06:39:38.932505 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:38.925220 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e5ef051-999a-4ab8-bf50-568c17f68ce1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c0662f25-m6c5j\" (UID: \"5e5ef051-999a-4ab8-bf50-568c17f68ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-m6c5j" Nov 01 06:39:38.940502 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:38.936608 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e5ef051-999a-4ab8-bf50-568c17f68ce1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c0662f25-m6c5j\" (UID: \"5e5ef051-999a-4ab8-bf50-568c17f68ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-m6c5j" Nov 01 06:39:38.961505 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:38.944729 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp24x\" (UniqueName: \"kubernetes.io/projected/5e5ef051-999a-4ab8-bf50-568c17f68ce1-kube-api-access-rp24x\") pod \"maas-api-revoke-keys-e2e-mt-c0662f25-m6c5j\" (UID: \"5e5ef051-999a-4ab8-bf50-568c17f68ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-m6c5j" Nov 01 06:39:38.986556 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:38.955546 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e5ef051-999a-4ab8-bf50-568c17f68ce1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c0662f25-m6c5j\" (UID: \"5e5ef051-999a-4ab8-bf50-568c17f68ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-m6c5j" Nov 01 06:39:38.987438 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:38.981217 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e5ef051-999a-4ab8-bf50-568c17f68ce1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c0662f25-m6c5j\" (UID: \"5e5ef051-999a-4ab8-bf50-568c17f68ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-m6c5j" Nov 01 06:39:39.036502 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:39.035576 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp24x\" (UniqueName: \"kubernetes.io/projected/5e5ef051-999a-4ab8-bf50-568c17f68ce1-kube-api-access-rp24x\") pod \"maas-api-revoke-keys-e2e-mt-c0662f25-m6c5j\" (UID: \"5e5ef051-999a-4ab8-bf50-568c17f68ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-m6c5j" Nov 01 06:39:39.070886 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:39.067497 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-m6c5j" Nov 01 06:39:39.653259 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:39.653232 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-m6c5j"] Nov 01 06:39:39.654173 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:39:39.654149 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e5ef051_999a_4ab8_bf50_568c17f68ce1.slice/crio-484dc97e78098c7147ab3f6974df9957c6093339e20fc614b0fb1cf4e48e27dc WatchSource:0}: Error finding container 484dc97e78098c7147ab3f6974df9957c6093339e20fc614b0fb1cf4e48e27dc: Status 404 returned error can't find the container with id 484dc97e78098c7147ab3f6974df9957c6093339e20fc614b0fb1cf4e48e27dc Nov 01 06:39:39.974675 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:39:39.974637 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:39:40.610939 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:40.610905 5040 generic.go:356] "Generic (PLEG): container finished" podID="5e5ef051-999a-4ab8-bf50-568c17f68ce1" containerID="15d7a767d828caf0eb37f19b4e6b71c38005a986bb70883c285b08ae029878b7" exitCode=0 Nov 01 06:39:40.611039 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:40.610985 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-m6c5j" event={"ID":"5e5ef051-999a-4ab8-bf50-568c17f68ce1","Type":"ContainerDied","Data":"15d7a767d828caf0eb37f19b4e6b71c38005a986bb70883c285b08ae029878b7"} Nov 01 06:39:40.611088 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:40.611071 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-m6c5j" event={"ID":"5e5ef051-999a-4ab8-bf50-568c17f68ce1","Type":"ContainerStarted","Data":"484dc97e78098c7147ab3f6974df9957c6093339e20fc614b0fb1cf4e48e27dc"} Nov 01 06:39:41.771294 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:41.771264 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-m6c5j" Nov 01 06:39:41.800763 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:41.800736 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e5ef051-999a-4ab8-bf50-568c17f68ce1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e5ef051-999a-4ab8-bf50-568c17f68ce1-maas-api-openshift-ca\") pod \"5e5ef051-999a-4ab8-bf50-568c17f68ce1\" (UID: \"5e5ef051-999a-4ab8-bf50-568c17f68ce1\") " Nov 01 06:39:41.800913 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:41.800833 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e5ef051-999a-4ab8-bf50-568c17f68ce1-kube-api-access-rp24x\" (UniqueName: \"kubernetes.io/projected/5e5ef051-999a-4ab8-bf50-568c17f68ce1-kube-api-access-rp24x\") pod \"5e5ef051-999a-4ab8-bf50-568c17f68ce1\" (UID: \"5e5ef051-999a-4ab8-bf50-568c17f68ce1\") " Nov 01 06:39:41.800913 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:41.800896 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e5ef051-999a-4ab8-bf50-568c17f68ce1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e5ef051-999a-4ab8-bf50-568c17f68ce1-maas-api-xks-ca\") pod \"5e5ef051-999a-4ab8-bf50-568c17f68ce1\" (UID: \"5e5ef051-999a-4ab8-bf50-568c17f68ce1\") " Nov 01 06:39:41.801092 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:41.801070 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e5ef051-999a-4ab8-bf50-568c17f68ce1-maas-api-openshift-ca" pod "5e5ef051-999a-4ab8-bf50-568c17f68ce1" (UID: "5e5ef051-999a-4ab8-bf50-568c17f68ce1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:39:41.801273 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:41.801254 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e5ef051-999a-4ab8-bf50-568c17f68ce1-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:39:41.802959 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:41.802936 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e5ef051-999a-4ab8-bf50-568c17f68ce1-kube-api-access-rp24x" pod "5e5ef051-999a-4ab8-bf50-568c17f68ce1" (UID: "5e5ef051-999a-4ab8-bf50-568c17f68ce1"). InnerVolumeSpecName "kube-api-access-rp24x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:39:41.803058 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:41.802941 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e5ef051-999a-4ab8-bf50-568c17f68ce1-maas-api-xks-ca" pod "5e5ef051-999a-4ab8-bf50-568c17f68ce1" (UID: "5e5ef051-999a-4ab8-bf50-568c17f68ce1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:39:41.902149 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:41.902076 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp24x\" (UniqueName: \"kubernetes.io/projected/5e5ef051-999a-4ab8-bf50-568c17f68ce1-kube-api-access-rp24x\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:39:41.902149 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:41.902107 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e5ef051-999a-4ab8-bf50-568c17f68ce1-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:39:42.626847 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:42.626814 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-m6c5j" Nov 01 06:39:42.627021 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:42.626844 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-m6c5j" event={"ID":"5e5ef051-999a-4ab8-bf50-568c17f68ce1","Type":"ContainerDied","Data":"484dc97e78098c7147ab3f6974df9957c6093339e20fc614b0fb1cf4e48e27dc"} Nov 01 06:39:42.627021 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:42.626892 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="484dc97e78098c7147ab3f6974df9957c6093339e20fc614b0fb1cf4e48e27dc" Nov 01 06:39:55.165934 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:55.165895 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-m6c5j"] Nov 01 06:39:55.216446 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:55.216409 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0662f25-m6c5j"] Nov 01 06:39:55.393238 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:55.393198 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn"] Nov 01 06:39:55.393550 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:55.393516 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" podUID="850bc9c2-fdcd-4a05-b7dc-1f04ea59f821" containerName="istio-proxy" containerID="cri-o://812ea0d4f5cfd3b440e1553c372b5a8cf350d484522a4897272377d1cde51d08" gracePeriod=30 Nov 01 06:39:55.418830 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:55.418744 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz"] Nov 01 06:39:55.419976 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:55.419920 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" podUID="1978f64d-b811-47bf-862a-aacbde03b6e9" containerName="istio-proxy" containerID="cri-o://390ee69e7c4e7e5b2acba384f2aa0c5cbd6dba9aa60a046b0bf23504347080a7" gracePeriod=30 Nov 01 06:39:55.428525 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:55.428466 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc"] Nov 01 06:39:55.428704 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:55.428683 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" podUID="72fe0dc9-1d5e-4be1-a48c-51af8c3e578c" containerName="istio-proxy" containerID="cri-o://ff39e9bd0769289400ca9eab27647ddc6af1840776b133a1bac7f5ee88a92bbe" gracePeriod=30 Nov 01 06:39:55.442872 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:55.442843 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh"] Nov 01 06:39:55.443041 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:55.443022 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" podUID="952eafb2-b8bc-457d-9bd6-4560f6176b74" containerName="istio-proxy" containerID="cri-o://fbbde0dd042ee7e3ed0d145a5bf4416e5e264307bedf4b951e88a75bb70c7a94" gracePeriod=30 Nov 01 06:39:55.519505 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:39:55.519446 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e5ef051-999a-4ab8-bf50-568c17f68ce1" path="/var/lib/kubelet/pods/5e5ef051-999a-4ab8-bf50-568c17f68ce1/volumes" Nov 01 06:40:00.740394 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.740362 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" Nov 01 06:40:00.744194 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.744166 5040 generic.go:356] "Generic (PLEG): container finished" podID="952eafb2-b8bc-457d-9bd6-4560f6176b74" containerID="fbbde0dd042ee7e3ed0d145a5bf4416e5e264307bedf4b951e88a75bb70c7a94" exitCode=0 Nov 01 06:40:00.744352 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.744269 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" event={"ID":"952eafb2-b8bc-457d-9bd6-4560f6176b74","Type":"ContainerDied","Data":"fbbde0dd042ee7e3ed0d145a5bf4416e5e264307bedf4b951e88a75bb70c7a94"} Nov 01 06:40:00.746882 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.746854 5040 generic.go:356] "Generic (PLEG): container finished" podID="1978f64d-b811-47bf-862a-aacbde03b6e9" containerID="390ee69e7c4e7e5b2acba384f2aa0c5cbd6dba9aa60a046b0bf23504347080a7" exitCode=0 Nov 01 06:40:00.747041 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.747024 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" event={"ID":"1978f64d-b811-47bf-862a-aacbde03b6e9","Type":"ContainerDied","Data":"390ee69e7c4e7e5b2acba384f2aa0c5cbd6dba9aa60a046b0bf23504347080a7"} Nov 01 06:40:00.747103 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.747080 5040 scope.go:122] "RemoveContainer" containerID="d007250bc3884dd71ace59931222b603f24b26e9f439455797992cd76c981cc4" Nov 01 06:40:00.749156 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.749129 5040 generic.go:356] "Generic (PLEG): container finished" podID="72fe0dc9-1d5e-4be1-a48c-51af8c3e578c" containerID="ff39e9bd0769289400ca9eab27647ddc6af1840776b133a1bac7f5ee88a92bbe" exitCode=0 Nov 01 06:40:00.749262 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.749229 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" event={"ID":"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c","Type":"ContainerDied","Data":"ff39e9bd0769289400ca9eab27647ddc6af1840776b133a1bac7f5ee88a92bbe"} Nov 01 06:40:00.750692 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.750669 5040 generic.go:356] "Generic (PLEG): container finished" podID="850bc9c2-fdcd-4a05-b7dc-1f04ea59f821" containerID="812ea0d4f5cfd3b440e1553c372b5a8cf350d484522a4897272377d1cde51d08" exitCode=0 Nov 01 06:40:00.750813 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.750742 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" Nov 01 06:40:00.750813 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.750765 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" event={"ID":"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821","Type":"ContainerDied","Data":"812ea0d4f5cfd3b440e1553c372b5a8cf350d484522a4897272377d1cde51d08"} Nov 01 06:40:00.750813 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.750799 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn" event={"ID":"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821","Type":"ContainerDied","Data":"26b1bd34e621ceec582128e263928ea38227b664d2d88a38f5d14849729b4687"} Nov 01 06:40:00.763695 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.763669 5040 scope.go:122] "RemoveContainer" containerID="812ea0d4f5cfd3b440e1553c372b5a8cf350d484522a4897272377d1cde51d08" Nov 01 06:40:00.774881 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.774858 5040 scope.go:122] "RemoveContainer" containerID="812ea0d4f5cfd3b440e1553c372b5a8cf350d484522a4897272377d1cde51d08" Nov 01 06:40:00.775208 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:40:00.775177 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"812ea0d4f5cfd3b440e1553c372b5a8cf350d484522a4897272377d1cde51d08\": container with ID starting with 812ea0d4f5cfd3b440e1553c372b5a8cf350d484522a4897272377d1cde51d08 not found: ID does not exist" containerID="812ea0d4f5cfd3b440e1553c372b5a8cf350d484522a4897272377d1cde51d08" Nov 01 06:40:00.775305 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.775222 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"812ea0d4f5cfd3b440e1553c372b5a8cf350d484522a4897272377d1cde51d08"} err="failed to get container status \"812ea0d4f5cfd3b440e1553c372b5a8cf350d484522a4897272377d1cde51d08\": rpc error: code = NotFound desc = could not find container \"812ea0d4f5cfd3b440e1553c372b5a8cf350d484522a4897272377d1cde51d08\": container with ID starting with 812ea0d4f5cfd3b440e1553c372b5a8cf350d484522a4897272377d1cde51d08 not found: ID does not exist" Nov 01 06:40:00.828648 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.828622 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-istio-podinfo\") pod \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\" (UID: \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\") " Nov 01 06:40:00.828822 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.828807 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-workload-certs\") pod \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\" (UID: \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\") " Nov 01 06:40:00.828901 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.828837 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-istio-data\") pod \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\" (UID: \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\") " Nov 01 06:40:00.828901 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.828892 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-istiod-ca-cert\") pod \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\" (UID: \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\") " Nov 01 06:40:00.829011 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.828938 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-credential-socket\") pod \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\" (UID: \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\") " Nov 01 06:40:00.829011 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.828988 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-istio-envoy\") pod \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\" (UID: \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\") " Nov 01 06:40:00.829120 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.829028 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-workload-socket\") pod \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\" (UID: \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\") " Nov 01 06:40:00.829120 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.829073 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-workload-certs" pod "850bc9c2-fdcd-4a05-b7dc-1f04ea59f821" (UID: "850bc9c2-fdcd-4a05-b7dc-1f04ea59f821"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:00.829120 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.829103 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-kube-api-access-rhk2s\" (UniqueName: \"kubernetes.io/projected/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-kube-api-access-rhk2s\") pod \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\" (UID: \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\") " Nov 01 06:40:00.829290 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.829148 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-istio-token\" (UniqueName: \"kubernetes.io/projected/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-istio-token\") pod \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\" (UID: \"850bc9c2-fdcd-4a05-b7dc-1f04ea59f821\") " Nov 01 06:40:00.829290 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.829263 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-credential-socket" pod "850bc9c2-fdcd-4a05-b7dc-1f04ea59f821" (UID: "850bc9c2-fdcd-4a05-b7dc-1f04ea59f821"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:00.829539 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.829511 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-istio-data" pod "850bc9c2-fdcd-4a05-b7dc-1f04ea59f821" (UID: "850bc9c2-fdcd-4a05-b7dc-1f04ea59f821"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:00.829680 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.829656 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-istiod-ca-cert" pod "850bc9c2-fdcd-4a05-b7dc-1f04ea59f821" (UID: "850bc9c2-fdcd-4a05-b7dc-1f04ea59f821"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:40:00.829789 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.829659 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:00.829789 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.829704 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:00.829789 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.829720 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:00.829789 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.829714 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-workload-socket" pod "850bc9c2-fdcd-4a05-b7dc-1f04ea59f821" (UID: "850bc9c2-fdcd-4a05-b7dc-1f04ea59f821"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:00.831415 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.831356 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" Nov 01 06:40:00.831836 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.831803 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-istio-token" pod "850bc9c2-fdcd-4a05-b7dc-1f04ea59f821" (UID: "850bc9c2-fdcd-4a05-b7dc-1f04ea59f821"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:40:00.832056 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.832038 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-istio-podinfo" pod "850bc9c2-fdcd-4a05-b7dc-1f04ea59f821" (UID: "850bc9c2-fdcd-4a05-b7dc-1f04ea59f821"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:40:00.832295 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.832276 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-kube-api-access-rhk2s" pod "850bc9c2-fdcd-4a05-b7dc-1f04ea59f821" (UID: "850bc9c2-fdcd-4a05-b7dc-1f04ea59f821"). InnerVolumeSpecName "kube-api-access-rhk2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:40:00.833831 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.833812 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" Nov 01 06:40:00.835182 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.835159 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-istio-envoy" pod "850bc9c2-fdcd-4a05-b7dc-1f04ea59f821" (UID: "850bc9c2-fdcd-4a05-b7dc-1f04ea59f821"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:00.837718 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.837702 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:40:00.930821 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.930789 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/952eafb2-b8bc-457d-9bd6-4560f6176b74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/952eafb2-b8bc-457d-9bd6-4560f6176b74-workload-certs\") pod \"952eafb2-b8bc-457d-9bd6-4560f6176b74\" (UID: \"952eafb2-b8bc-457d-9bd6-4560f6176b74\") " Nov 01 06:40:00.930821 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.930821 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/952eafb2-b8bc-457d-9bd6-4560f6176b74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/952eafb2-b8bc-457d-9bd6-4560f6176b74-istio-envoy\") pod \"952eafb2-b8bc-457d-9bd6-4560f6176b74\" (UID: \"952eafb2-b8bc-457d-9bd6-4560f6176b74\") " Nov 01 06:40:00.930821 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.930840 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-workload-socket\") pod \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\" (UID: \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\") " Nov 01 06:40:00.931154 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.930863 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-istio-envoy\") pod \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\" (UID: \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\") " Nov 01 06:40:00.931154 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.930887 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-workload-certs\") pod \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\" (UID: \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\") " Nov 01 06:40:00.931154 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.930956 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/952eafb2-b8bc-457d-9bd6-4560f6176b74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/952eafb2-b8bc-457d-9bd6-4560f6176b74-credential-socket\") pod \"952eafb2-b8bc-457d-9bd6-4560f6176b74\" (UID: \"952eafb2-b8bc-457d-9bd6-4560f6176b74\") " Nov 01 06:40:00.931154 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.930999 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/952eafb2-b8bc-457d-9bd6-4560f6176b74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/952eafb2-b8bc-457d-9bd6-4560f6176b74-istio-podinfo\") pod \"952eafb2-b8bc-457d-9bd6-4560f6176b74\" (UID: \"952eafb2-b8bc-457d-9bd6-4560f6176b74\") " Nov 01 06:40:00.931154 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.931031 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-credential-socket\") pod \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\" (UID: \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\") " Nov 01 06:40:00.931154 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.931069 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/952eafb2-b8bc-457d-9bd6-4560f6176b74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/952eafb2-b8bc-457d-9bd6-4560f6176b74-workload-socket\") pod \"952eafb2-b8bc-457d-9bd6-4560f6176b74\" (UID: \"952eafb2-b8bc-457d-9bd6-4560f6176b74\") " Nov 01 06:40:00.931154 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.931132 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1978f64d-b811-47bf-862a-aacbde03b6e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1978f64d-b811-47bf-862a-aacbde03b6e9-workload-certs\") pod \"1978f64d-b811-47bf-862a-aacbde03b6e9\" (UID: \"1978f64d-b811-47bf-862a-aacbde03b6e9\") " Nov 01 06:40:00.931154 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.931144 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952eafb2-b8bc-457d-9bd6-4560f6176b74-workload-certs" pod "952eafb2-b8bc-457d-9bd6-4560f6176b74" (UID: "952eafb2-b8bc-457d-9bd6-4560f6176b74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:00.931625 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.931142 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-workload-socket" pod "72fe0dc9-1d5e-4be1-a48c-51af8c3e578c" (UID: "72fe0dc9-1d5e-4be1-a48c-51af8c3e578c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:00.931625 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.931165 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-istio-data\") pod \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\" (UID: \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\") " Nov 01 06:40:00.931625 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.931237 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/952eafb2-b8bc-457d-9bd6-4560f6176b74-kube-api-access-xkh5t\" (UniqueName: \"kubernetes.io/projected/952eafb2-b8bc-457d-9bd6-4560f6176b74-kube-api-access-xkh5t\") pod \"952eafb2-b8bc-457d-9bd6-4560f6176b74\" (UID: \"952eafb2-b8bc-457d-9bd6-4560f6176b74\") " Nov 01 06:40:00.931625 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.931241 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-workload-certs" pod "72fe0dc9-1d5e-4be1-a48c-51af8c3e578c" (UID: "72fe0dc9-1d5e-4be1-a48c-51af8c3e578c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:00.931625 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.931289 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/952eafb2-b8bc-457d-9bd6-4560f6176b74-istio-token\" (UniqueName: \"kubernetes.io/projected/952eafb2-b8bc-457d-9bd6-4560f6176b74-istio-token\") pod \"952eafb2-b8bc-457d-9bd6-4560f6176b74\" (UID: \"952eafb2-b8bc-457d-9bd6-4560f6176b74\") " Nov 01 06:40:00.931625 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.931316 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1978f64d-b811-47bf-862a-aacbde03b6e9-istio-token\" (UniqueName: \"kubernetes.io/projected/1978f64d-b811-47bf-862a-aacbde03b6e9-istio-token\") pod \"1978f64d-b811-47bf-862a-aacbde03b6e9\" (UID: \"1978f64d-b811-47bf-862a-aacbde03b6e9\") " Nov 01 06:40:00.931625 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.931351 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-istio-data" pod "72fe0dc9-1d5e-4be1-a48c-51af8c3e578c" (UID: "72fe0dc9-1d5e-4be1-a48c-51af8c3e578c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:00.931625 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.931373 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/952eafb2-b8bc-457d-9bd6-4560f6176b74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/952eafb2-b8bc-457d-9bd6-4560f6176b74-istio-data\") pod \"952eafb2-b8bc-457d-9bd6-4560f6176b74\" (UID: \"952eafb2-b8bc-457d-9bd6-4560f6176b74\") " Nov 01 06:40:00.931625 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.931412 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-istio-token\" (UniqueName: \"kubernetes.io/projected/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-istio-token\") pod \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\" (UID: \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\") " Nov 01 06:40:00.931625 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.931462 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1978f64d-b811-47bf-862a-aacbde03b6e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1978f64d-b811-47bf-862a-aacbde03b6e9-istio-podinfo\") pod \"1978f64d-b811-47bf-862a-aacbde03b6e9\" (UID: \"1978f64d-b811-47bf-862a-aacbde03b6e9\") " Nov 01 06:40:00.931625 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.931508 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-istio-podinfo\") pod \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\" (UID: \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\") " Nov 01 06:40:00.931625 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.931545 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952eafb2-b8bc-457d-9bd6-4560f6176b74-workload-socket" pod "952eafb2-b8bc-457d-9bd6-4560f6176b74" (UID: "952eafb2-b8bc-457d-9bd6-4560f6176b74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:00.931625 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.931572 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1978f64d-b811-47bf-862a-aacbde03b6e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1978f64d-b811-47bf-862a-aacbde03b6e9-istio-envoy\") pod \"1978f64d-b811-47bf-862a-aacbde03b6e9\" (UID: \"1978f64d-b811-47bf-862a-aacbde03b6e9\") " Nov 01 06:40:00.931625 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.931574 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1978f64d-b811-47bf-862a-aacbde03b6e9-workload-certs" pod "1978f64d-b811-47bf-862a-aacbde03b6e9" (UID: "1978f64d-b811-47bf-862a-aacbde03b6e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:00.931625 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.931599 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1978f64d-b811-47bf-862a-aacbde03b6e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1978f64d-b811-47bf-862a-aacbde03b6e9-credential-socket\") pod \"1978f64d-b811-47bf-862a-aacbde03b6e9\" (UID: \"1978f64d-b811-47bf-862a-aacbde03b6e9\") " Nov 01 06:40:00.932439 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.931638 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-kube-api-access-n26vp\" (UniqueName: \"kubernetes.io/projected/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-kube-api-access-n26vp\") pod \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\" (UID: \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\") " Nov 01 06:40:00.932439 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.931678 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-istiod-ca-cert\") pod \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\" (UID: \"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c\") " Nov 01 06:40:00.932439 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.931765 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952eafb2-b8bc-457d-9bd6-4560f6176b74-credential-socket" pod "952eafb2-b8bc-457d-9bd6-4560f6176b74" (UID: "952eafb2-b8bc-457d-9bd6-4560f6176b74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:00.932439 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.931776 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952eafb2-b8bc-457d-9bd6-4560f6176b74-istio-data" pod "952eafb2-b8bc-457d-9bd6-4560f6176b74" (UID: "952eafb2-b8bc-457d-9bd6-4560f6176b74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:00.932439 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.931769 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-credential-socket" pod "72fe0dc9-1d5e-4be1-a48c-51af8c3e578c" (UID: "72fe0dc9-1d5e-4be1-a48c-51af8c3e578c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:00.932439 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.931705 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1978f64d-b811-47bf-862a-aacbde03b6e9-kube-api-access-bd8zz\" (UniqueName: \"kubernetes.io/projected/1978f64d-b811-47bf-862a-aacbde03b6e9-kube-api-access-bd8zz\") pod \"1978f64d-b811-47bf-862a-aacbde03b6e9\" (UID: \"1978f64d-b811-47bf-862a-aacbde03b6e9\") " Nov 01 06:40:00.932439 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.932086 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/952eafb2-b8bc-457d-9bd6-4560f6176b74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/952eafb2-b8bc-457d-9bd6-4560f6176b74-istiod-ca-cert\") pod \"952eafb2-b8bc-457d-9bd6-4560f6176b74\" (UID: \"952eafb2-b8bc-457d-9bd6-4560f6176b74\") " Nov 01 06:40:00.932439 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.932141 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1978f64d-b811-47bf-862a-aacbde03b6e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1978f64d-b811-47bf-862a-aacbde03b6e9-istiod-ca-cert\") pod \"1978f64d-b811-47bf-862a-aacbde03b6e9\" (UID: \"1978f64d-b811-47bf-862a-aacbde03b6e9\") " Nov 01 06:40:00.932439 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.932177 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1978f64d-b811-47bf-862a-aacbde03b6e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1978f64d-b811-47bf-862a-aacbde03b6e9-istio-data\") pod \"1978f64d-b811-47bf-862a-aacbde03b6e9\" (UID: \"1978f64d-b811-47bf-862a-aacbde03b6e9\") " Nov 01 06:40:00.932439 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.932215 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1978f64d-b811-47bf-862a-aacbde03b6e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1978f64d-b811-47bf-862a-aacbde03b6e9-workload-socket\") pod \"1978f64d-b811-47bf-862a-aacbde03b6e9\" (UID: \"1978f64d-b811-47bf-862a-aacbde03b6e9\") " Nov 01 06:40:00.932439 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.932405 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1978f64d-b811-47bf-862a-aacbde03b6e9-credential-socket" pod "1978f64d-b811-47bf-862a-aacbde03b6e9" (UID: "1978f64d-b811-47bf-862a-aacbde03b6e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:00.933045 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.932639 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1978f64d-b811-47bf-862a-aacbde03b6e9-workload-socket" pod "1978f64d-b811-47bf-862a-aacbde03b6e9" (UID: "1978f64d-b811-47bf-862a-aacbde03b6e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:00.933045 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.932864 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:00.933045 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.932884 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:00.933045 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.932885 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1978f64d-b811-47bf-862a-aacbde03b6e9-istiod-ca-cert" pod "1978f64d-b811-47bf-862a-aacbde03b6e9" (UID: "1978f64d-b811-47bf-862a-aacbde03b6e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:40:00.933045 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.932897 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1978f64d-b811-47bf-862a-aacbde03b6e9-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:00.933045 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.932912 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhk2s\" (UniqueName: \"kubernetes.io/projected/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-kube-api-access-rhk2s\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:00.933045 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.932927 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:00.933045 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.932941 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/952eafb2-b8bc-457d-9bd6-4560f6176b74-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:00.933045 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.932955 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:00.933045 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.932971 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:00.933045 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.932985 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:00.933045 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.933000 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/952eafb2-b8bc-457d-9bd6-4560f6176b74-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:00.933045 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.933015 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:00.933045 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.933030 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/952eafb2-b8bc-457d-9bd6-4560f6176b74-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:00.933045 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.933046 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1978f64d-b811-47bf-862a-aacbde03b6e9-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:00.933890 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.933060 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:00.933890 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.933074 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/952eafb2-b8bc-457d-9bd6-4560f6176b74-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:00.933890 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.933090 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:00.933890 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.933115 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1978f64d-b811-47bf-862a-aacbde03b6e9-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:00.933890 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.933246 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/952eafb2-b8bc-457d-9bd6-4560f6176b74-istiod-ca-cert" pod "952eafb2-b8bc-457d-9bd6-4560f6176b74" (UID: "952eafb2-b8bc-457d-9bd6-4560f6176b74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:40:00.933890 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.933344 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1978f64d-b811-47bf-862a-aacbde03b6e9-istio-data" pod "1978f64d-b811-47bf-862a-aacbde03b6e9" (UID: "1978f64d-b811-47bf-862a-aacbde03b6e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:00.934232 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.934044 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-istiod-ca-cert" pod "72fe0dc9-1d5e-4be1-a48c-51af8c3e578c" (UID: "72fe0dc9-1d5e-4be1-a48c-51af8c3e578c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:40:00.934232 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.934114 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/952eafb2-b8bc-457d-9bd6-4560f6176b74-istio-podinfo" pod "952eafb2-b8bc-457d-9bd6-4560f6176b74" (UID: "952eafb2-b8bc-457d-9bd6-4560f6176b74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:40:00.936252 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.936224 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-kube-api-access-n26vp" pod "72fe0dc9-1d5e-4be1-a48c-51af8c3e578c" (UID: "72fe0dc9-1d5e-4be1-a48c-51af8c3e578c"). InnerVolumeSpecName "kube-api-access-n26vp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:40:00.936465 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.936435 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1978f64d-b811-47bf-862a-aacbde03b6e9-istio-envoy" pod "1978f64d-b811-47bf-862a-aacbde03b6e9" (UID: "1978f64d-b811-47bf-862a-aacbde03b6e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:00.936873 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.936837 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-istio-token" pod "72fe0dc9-1d5e-4be1-a48c-51af8c3e578c" (UID: "72fe0dc9-1d5e-4be1-a48c-51af8c3e578c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:40:00.936979 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.936919 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/952eafb2-b8bc-457d-9bd6-4560f6176b74-kube-api-access-xkh5t" pod "952eafb2-b8bc-457d-9bd6-4560f6176b74" (UID: "952eafb2-b8bc-457d-9bd6-4560f6176b74"). InnerVolumeSpecName "kube-api-access-xkh5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:40:00.937048 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.936970 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1978f64d-b811-47bf-862a-aacbde03b6e9-kube-api-access-bd8zz" pod "1978f64d-b811-47bf-862a-aacbde03b6e9" (UID: "1978f64d-b811-47bf-862a-aacbde03b6e9"). InnerVolumeSpecName "kube-api-access-bd8zz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:40:00.937114 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.937046 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952eafb2-b8bc-457d-9bd6-4560f6176b74-istio-envoy" pod "952eafb2-b8bc-457d-9bd6-4560f6176b74" (UID: "952eafb2-b8bc-457d-9bd6-4560f6176b74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:00.937114 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.937073 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1978f64d-b811-47bf-862a-aacbde03b6e9-istio-token" pod "1978f64d-b811-47bf-862a-aacbde03b6e9" (UID: "1978f64d-b811-47bf-862a-aacbde03b6e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:40:00.937368 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.937343 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/952eafb2-b8bc-457d-9bd6-4560f6176b74-istio-token" pod "952eafb2-b8bc-457d-9bd6-4560f6176b74" (UID: "952eafb2-b8bc-457d-9bd6-4560f6176b74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:40:00.937659 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.937626 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-istio-envoy" pod "72fe0dc9-1d5e-4be1-a48c-51af8c3e578c" (UID: "72fe0dc9-1d5e-4be1-a48c-51af8c3e578c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:00.938312 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.938285 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1978f64d-b811-47bf-862a-aacbde03b6e9-istio-podinfo" pod "1978f64d-b811-47bf-862a-aacbde03b6e9" (UID: "1978f64d-b811-47bf-862a-aacbde03b6e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:40:00.938555 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:00.938463 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-istio-podinfo" pod "72fe0dc9-1d5e-4be1-a48c-51af8c3e578c" (UID: "72fe0dc9-1d5e-4be1-a48c-51af8c3e578c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:40:01.034046 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:01.034001 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:01.034046 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:01.034036 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1978f64d-b811-47bf-862a-aacbde03b6e9-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:01.034046 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:01.034046 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:01.034046 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:01.034055 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1978f64d-b811-47bf-862a-aacbde03b6e9-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:01.034435 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:01.034064 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n26vp\" (UniqueName: \"kubernetes.io/projected/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-kube-api-access-n26vp\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:01.034435 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:01.034073 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:01.034435 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:01.034082 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd8zz\" (UniqueName: \"kubernetes.io/projected/1978f64d-b811-47bf-862a-aacbde03b6e9-kube-api-access-bd8zz\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:01.034435 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:01.034091 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/952eafb2-b8bc-457d-9bd6-4560f6176b74-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:01.034435 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:01.034099 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1978f64d-b811-47bf-862a-aacbde03b6e9-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:01.034435 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:01.034107 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1978f64d-b811-47bf-862a-aacbde03b6e9-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:01.034435 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:01.034116 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/952eafb2-b8bc-457d-9bd6-4560f6176b74-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:01.034435 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:01.034126 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:01.034435 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:01.034135 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/952eafb2-b8bc-457d-9bd6-4560f6176b74-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:01.034435 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:01.034145 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkh5t\" (UniqueName: \"kubernetes.io/projected/952eafb2-b8bc-457d-9bd6-4560f6176b74-kube-api-access-xkh5t\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:01.034435 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:01.034153 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/952eafb2-b8bc-457d-9bd6-4560f6176b74-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:01.034435 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:01.034161 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1978f64d-b811-47bf-862a-aacbde03b6e9-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:01.452885 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:01.452842 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn"] Nov 01 06:40:01.461702 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:01.461674 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-s6nwn"] Nov 01 06:40:01.514929 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:01.514897 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="850bc9c2-fdcd-4a05-b7dc-1f04ea59f821" path="/var/lib/kubelet/pods/850bc9c2-fdcd-4a05-b7dc-1f04ea59f821/volumes" Nov 01 06:40:01.760189 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:01.760151 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" Nov 01 06:40:01.760189 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:01.760170 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh" event={"ID":"952eafb2-b8bc-457d-9bd6-4560f6176b74","Type":"ContainerDied","Data":"53935ec2522d5353a0beba50ce3c68ad578cc69c15135de296bedeb1d9a7cd47"} Nov 01 06:40:01.760724 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:01.760240 5040 scope.go:122] "RemoveContainer" containerID="fbbde0dd042ee7e3ed0d145a5bf4416e5e264307bedf4b951e88a75bb70c7a94" Nov 01 06:40:01.761704 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:01.761688 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" Nov 01 06:40:01.761799 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:01.761682 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz" event={"ID":"1978f64d-b811-47bf-862a-aacbde03b6e9","Type":"ContainerDied","Data":"c4411af092eccd7b4265f3e4f7f8cb6890c0898681e8f4b4b1fe6b9ba62f247b"} Nov 01 06:40:01.763529 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:01.763474 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" event={"ID":"72fe0dc9-1d5e-4be1-a48c-51af8c3e578c","Type":"ContainerDied","Data":"175ca02a83930a752d406881a83e5e6033c65a4bf4b5ccbe9718a09111e0f1f0"} Nov 01 06:40:01.763666 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:01.763529 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc" Nov 01 06:40:01.774216 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:01.774192 5040 scope.go:122] "RemoveContainer" containerID="390ee69e7c4e7e5b2acba384f2aa0c5cbd6dba9aa60a046b0bf23504347080a7" Nov 01 06:40:01.783940 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:01.783924 5040 scope.go:122] "RemoveContainer" containerID="ff39e9bd0769289400ca9eab27647ddc6af1840776b133a1bac7f5ee88a92bbe" Nov 01 06:40:01.930416 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:01.930383 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz"] Nov 01 06:40:01.936299 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:01.936273 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-tskhz"] Nov 01 06:40:02.047996 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.047902 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh"] Nov 01 06:40:02.051895 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.051869 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-hp8zh"] Nov 01 06:40:02.162976 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.162942 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc"] Nov 01 06:40:02.169462 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.169431 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-w44jc"] Nov 01 06:40:02.348844 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.348756 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76d2a29e-xrmw7"] Nov 01 06:40:02.349238 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.349223 5040 cpu_manager.go:414] "containerMap: removing container" podUID="850bc9c2-fdcd-4a05-b7dc-1f04ea59f821" containerName="istio-proxy" Nov 01 06:40:02.349238 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.349237 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="850bc9c2-fdcd-4a05-b7dc-1f04ea59f821" containerName="istio-proxy" Nov 01 06:40:02.349365 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.349249 5040 cpu_manager.go:414] "containerMap: removing container" podUID="1978f64d-b811-47bf-862a-aacbde03b6e9" containerName="istio-proxy" Nov 01 06:40:02.349365 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.349255 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1978f64d-b811-47bf-862a-aacbde03b6e9" containerName="istio-proxy" Nov 01 06:40:02.349365 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.349269 5040 cpu_manager.go:414] "containerMap: removing container" podUID="1978f64d-b811-47bf-862a-aacbde03b6e9" containerName="istio-proxy" Nov 01 06:40:02.349365 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.349275 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1978f64d-b811-47bf-862a-aacbde03b6e9" containerName="istio-proxy" Nov 01 06:40:02.349365 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.349283 5040 cpu_manager.go:414] "containerMap: removing container" podUID="72fe0dc9-1d5e-4be1-a48c-51af8c3e578c" containerName="istio-proxy" Nov 01 06:40:02.349365 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.349287 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72fe0dc9-1d5e-4be1-a48c-51af8c3e578c" containerName="istio-proxy" Nov 01 06:40:02.349365 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.349303 5040 cpu_manager.go:414] "containerMap: removing container" podUID="5e5ef051-999a-4ab8-bf50-568c17f68ce1" containerName="revoke-keys" Nov 01 06:40:02.349365 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.349309 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e5ef051-999a-4ab8-bf50-568c17f68ce1" containerName="revoke-keys" Nov 01 06:40:02.349365 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.349320 5040 cpu_manager.go:414] "containerMap: removing container" podUID="952eafb2-b8bc-457d-9bd6-4560f6176b74" containerName="istio-proxy" Nov 01 06:40:02.349365 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.349325 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="952eafb2-b8bc-457d-9bd6-4560f6176b74" containerName="istio-proxy" Nov 01 06:40:02.349809 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.349399 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="72fe0dc9-1d5e-4be1-a48c-51af8c3e578c" containerName="istio-proxy" Nov 01 06:40:02.349809 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.349411 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e5ef051-999a-4ab8-bf50-568c17f68ce1" containerName="revoke-keys" Nov 01 06:40:02.349809 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.349418 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="952eafb2-b8bc-457d-9bd6-4560f6176b74" containerName="istio-proxy" Nov 01 06:40:02.349809 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.349428 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="850bc9c2-fdcd-4a05-b7dc-1f04ea59f821" containerName="istio-proxy" Nov 01 06:40:02.349809 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.349433 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="1978f64d-b811-47bf-862a-aacbde03b6e9" containerName="istio-proxy" Nov 01 06:40:02.354269 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.354251 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76d2a29e-xrmw7" Nov 01 06:40:02.407668 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.407634 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:40:02.407668 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.407641 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:40:02.407919 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.407679 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:40:02.407919 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.407863 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:40:02.410449 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.410427 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76d2a29e-xrmw7"] Nov 01 06:40:02.449099 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.449060 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7csf\" (UniqueName: \"kubernetes.io/projected/2838d17e-42d0-4388-9e7b-8075d9b9503d-kube-api-access-j7csf\") pod \"maas-api-revoke-sub-76d2a29e-xrmw7\" (UID: \"2838d17e-42d0-4388-9e7b-8075d9b9503d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76d2a29e-xrmw7" Nov 01 06:40:02.449259 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.449213 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2838d17e-42d0-4388-9e7b-8075d9b9503d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76d2a29e-xrmw7\" (UID: \"2838d17e-42d0-4388-9e7b-8075d9b9503d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76d2a29e-xrmw7" Nov 01 06:40:02.449347 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.449309 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2838d17e-42d0-4388-9e7b-8075d9b9503d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76d2a29e-xrmw7\" (UID: \"2838d17e-42d0-4388-9e7b-8075d9b9503d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76d2a29e-xrmw7" Nov 01 06:40:02.550894 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.550859 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7csf\" (UniqueName: \"kubernetes.io/projected/2838d17e-42d0-4388-9e7b-8075d9b9503d-kube-api-access-j7csf\") pod \"maas-api-revoke-sub-76d2a29e-xrmw7\" (UID: \"2838d17e-42d0-4388-9e7b-8075d9b9503d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76d2a29e-xrmw7" Nov 01 06:40:02.551115 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.550965 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2838d17e-42d0-4388-9e7b-8075d9b9503d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76d2a29e-xrmw7\" (UID: \"2838d17e-42d0-4388-9e7b-8075d9b9503d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76d2a29e-xrmw7" Nov 01 06:40:02.551115 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.550993 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2838d17e-42d0-4388-9e7b-8075d9b9503d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76d2a29e-xrmw7\" (UID: \"2838d17e-42d0-4388-9e7b-8075d9b9503d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76d2a29e-xrmw7" Nov 01 06:40:02.551800 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.551779 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2838d17e-42d0-4388-9e7b-8075d9b9503d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76d2a29e-xrmw7\" (UID: \"2838d17e-42d0-4388-9e7b-8075d9b9503d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76d2a29e-xrmw7" Nov 01 06:40:02.553396 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.553379 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2838d17e-42d0-4388-9e7b-8075d9b9503d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76d2a29e-xrmw7\" (UID: \"2838d17e-42d0-4388-9e7b-8075d9b9503d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76d2a29e-xrmw7" Nov 01 06:40:02.613045 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.612968 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7csf\" (UniqueName: \"kubernetes.io/projected/2838d17e-42d0-4388-9e7b-8075d9b9503d-kube-api-access-j7csf\") pod \"maas-api-revoke-sub-76d2a29e-xrmw7\" (UID: \"2838d17e-42d0-4388-9e7b-8075d9b9503d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76d2a29e-xrmw7" Nov 01 06:40:02.668240 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:02.668202 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76d2a29e-xrmw7" Nov 01 06:40:03.042352 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:03.042325 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76d2a29e-xrmw7"] Nov 01 06:40:03.043215 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:40:03.043190 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2838d17e_42d0_4388_9e7b_8075d9b9503d.slice/crio-880a84cc69344f1c86314b860be64e86b1a1410130d27d9cab0167331fa2c1f5 WatchSource:0}: Error finding container 880a84cc69344f1c86314b860be64e86b1a1410130d27d9cab0167331fa2c1f5: Status 404 returned error can't find the container with id 880a84cc69344f1c86314b860be64e86b1a1410130d27d9cab0167331fa2c1f5 Nov 01 06:40:03.515946 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:03.515904 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1978f64d-b811-47bf-862a-aacbde03b6e9" path="/var/lib/kubelet/pods/1978f64d-b811-47bf-862a-aacbde03b6e9/volumes" Nov 01 06:40:03.516413 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:03.516396 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72fe0dc9-1d5e-4be1-a48c-51af8c3e578c" path="/var/lib/kubelet/pods/72fe0dc9-1d5e-4be1-a48c-51af8c3e578c/volumes" Nov 01 06:40:03.516870 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:03.516853 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="952eafb2-b8bc-457d-9bd6-4560f6176b74" path="/var/lib/kubelet/pods/952eafb2-b8bc-457d-9bd6-4560f6176b74/volumes" Nov 01 06:40:03.783445 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:03.783397 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76d2a29e-xrmw7" event={"ID":"2838d17e-42d0-4388-9e7b-8075d9b9503d","Type":"ContainerStarted","Data":"2d6da84bdf7a22bdf8f7b77777b5ae856960206b559bebb407ab95b5ddc7b560"} Nov 01 06:40:03.783832 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:03.783462 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76d2a29e-xrmw7" event={"ID":"2838d17e-42d0-4388-9e7b-8075d9b9503d","Type":"ContainerStarted","Data":"880a84cc69344f1c86314b860be64e86b1a1410130d27d9cab0167331fa2c1f5"} Nov 01 06:40:03.901905 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:03.901845 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76d2a29e-xrmw7" podStartSLOduration=1.9018324500000001 podStartE2EDuration="1.90183245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:40:03.900233898 +0000 UTC m=+3282.555351065" watchObservedRunningTime="2026-11-01 06:40:03.90183245 +0000 UTC m=+3282.556949616" Nov 01 06:40:04.791075 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:04.791040 5040 generic.go:356] "Generic (PLEG): container finished" podID="2838d17e-42d0-4388-9e7b-8075d9b9503d" containerID="2d6da84bdf7a22bdf8f7b77777b5ae856960206b559bebb407ab95b5ddc7b560" exitCode=0 Nov 01 06:40:04.791450 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:04.791117 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76d2a29e-xrmw7" event={"ID":"2838d17e-42d0-4388-9e7b-8075d9b9503d","Type":"ContainerDied","Data":"2d6da84bdf7a22bdf8f7b77777b5ae856960206b559bebb407ab95b5ddc7b560"} Nov 01 06:40:05.937213 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:05.937188 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76d2a29e-xrmw7" Nov 01 06:40:06.087187 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:06.087093 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2838d17e-42d0-4388-9e7b-8075d9b9503d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2838d17e-42d0-4388-9e7b-8075d9b9503d-maas-api-openshift-ca\") pod \"2838d17e-42d0-4388-9e7b-8075d9b9503d\" (UID: \"2838d17e-42d0-4388-9e7b-8075d9b9503d\") " Nov 01 06:40:06.087350 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:06.087234 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2838d17e-42d0-4388-9e7b-8075d9b9503d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2838d17e-42d0-4388-9e7b-8075d9b9503d-maas-api-xks-ca\") pod \"2838d17e-42d0-4388-9e7b-8075d9b9503d\" (UID: \"2838d17e-42d0-4388-9e7b-8075d9b9503d\") " Nov 01 06:40:06.087350 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:06.087313 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2838d17e-42d0-4388-9e7b-8075d9b9503d-kube-api-access-j7csf\" (UniqueName: \"kubernetes.io/projected/2838d17e-42d0-4388-9e7b-8075d9b9503d-kube-api-access-j7csf\") pod \"2838d17e-42d0-4388-9e7b-8075d9b9503d\" (UID: \"2838d17e-42d0-4388-9e7b-8075d9b9503d\") " Nov 01 06:40:06.087536 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:06.087512 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2838d17e-42d0-4388-9e7b-8075d9b9503d-maas-api-openshift-ca" pod "2838d17e-42d0-4388-9e7b-8075d9b9503d" (UID: "2838d17e-42d0-4388-9e7b-8075d9b9503d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:40:06.087789 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:06.087768 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2838d17e-42d0-4388-9e7b-8075d9b9503d-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:06.089383 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:06.089359 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2838d17e-42d0-4388-9e7b-8075d9b9503d-maas-api-xks-ca" pod "2838d17e-42d0-4388-9e7b-8075d9b9503d" (UID: "2838d17e-42d0-4388-9e7b-8075d9b9503d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:40:06.090520 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:06.090475 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2838d17e-42d0-4388-9e7b-8075d9b9503d-kube-api-access-j7csf" pod "2838d17e-42d0-4388-9e7b-8075d9b9503d" (UID: "2838d17e-42d0-4388-9e7b-8075d9b9503d"). InnerVolumeSpecName "kube-api-access-j7csf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:40:06.189131 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:06.189094 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2838d17e-42d0-4388-9e7b-8075d9b9503d-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:06.189131 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:06.189128 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7csf\" (UniqueName: \"kubernetes.io/projected/2838d17e-42d0-4388-9e7b-8075d9b9503d-kube-api-access-j7csf\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:06.282495 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:06.282453 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88b760ca-lfph6"] Nov 01 06:40:06.282987 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:06.282972 5040 cpu_manager.go:414] "containerMap: removing container" podUID="2838d17e-42d0-4388-9e7b-8075d9b9503d" containerName="revoke-keys" Nov 01 06:40:06.283052 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:06.282989 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2838d17e-42d0-4388-9e7b-8075d9b9503d" containerName="revoke-keys" Nov 01 06:40:06.283105 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:06.283098 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="2838d17e-42d0-4388-9e7b-8075d9b9503d" containerName="revoke-keys" Nov 01 06:40:06.283155 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:06.283109 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="1978f64d-b811-47bf-862a-aacbde03b6e9" containerName="istio-proxy" Nov 01 06:40:06.286667 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:06.286646 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b760ca-lfph6" Nov 01 06:40:06.345372 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:06.345294 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88b760ca-lfph6"] Nov 01 06:40:06.391071 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:06.391033 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d238dea-5ba1-4080-a7d9-74a985c08e99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88b760ca-lfph6\" (UID: \"0d238dea-5ba1-4080-a7d9-74a985c08e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b760ca-lfph6" Nov 01 06:40:06.391243 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:06.391143 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d238dea-5ba1-4080-a7d9-74a985c08e99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88b760ca-lfph6\" (UID: \"0d238dea-5ba1-4080-a7d9-74a985c08e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b760ca-lfph6" Nov 01 06:40:06.391243 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:06.391222 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp7n7\" (UniqueName: \"kubernetes.io/projected/0d238dea-5ba1-4080-a7d9-74a985c08e99-kube-api-access-qp7n7\") pod \"maas-api-revoke-sub-88b760ca-lfph6\" (UID: \"0d238dea-5ba1-4080-a7d9-74a985c08e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b760ca-lfph6" Nov 01 06:40:06.492906 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:06.492876 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp7n7\" (UniqueName: \"kubernetes.io/projected/0d238dea-5ba1-4080-a7d9-74a985c08e99-kube-api-access-qp7n7\") pod \"maas-api-revoke-sub-88b760ca-lfph6\" (UID: \"0d238dea-5ba1-4080-a7d9-74a985c08e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b760ca-lfph6" Nov 01 06:40:06.493085 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:06.492967 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d238dea-5ba1-4080-a7d9-74a985c08e99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88b760ca-lfph6\" (UID: \"0d238dea-5ba1-4080-a7d9-74a985c08e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b760ca-lfph6" Nov 01 06:40:06.493085 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:06.493016 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d238dea-5ba1-4080-a7d9-74a985c08e99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88b760ca-lfph6\" (UID: \"0d238dea-5ba1-4080-a7d9-74a985c08e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b760ca-lfph6" Nov 01 06:40:06.493720 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:06.493696 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d238dea-5ba1-4080-a7d9-74a985c08e99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88b760ca-lfph6\" (UID: \"0d238dea-5ba1-4080-a7d9-74a985c08e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b760ca-lfph6" Nov 01 06:40:06.495357 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:06.495334 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d238dea-5ba1-4080-a7d9-74a985c08e99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88b760ca-lfph6\" (UID: \"0d238dea-5ba1-4080-a7d9-74a985c08e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b760ca-lfph6" Nov 01 06:40:06.554367 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:06.554338 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp7n7\" (UniqueName: \"kubernetes.io/projected/0d238dea-5ba1-4080-a7d9-74a985c08e99-kube-api-access-qp7n7\") pod \"maas-api-revoke-sub-88b760ca-lfph6\" (UID: \"0d238dea-5ba1-4080-a7d9-74a985c08e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b760ca-lfph6" Nov 01 06:40:06.603328 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:06.603256 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b760ca-lfph6" Nov 01 06:40:06.805441 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:06.805413 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76d2a29e-xrmw7" Nov 01 06:40:06.805624 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:06.805438 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76d2a29e-xrmw7" event={"ID":"2838d17e-42d0-4388-9e7b-8075d9b9503d","Type":"ContainerDied","Data":"880a84cc69344f1c86314b860be64e86b1a1410130d27d9cab0167331fa2c1f5"} Nov 01 06:40:06.805624 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:06.805509 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="880a84cc69344f1c86314b860be64e86b1a1410130d27d9cab0167331fa2c1f5" Nov 01 06:40:06.977297 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:06.977268 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88b760ca-lfph6"] Nov 01 06:40:06.977882 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:40:06.977858 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d238dea_5ba1_4080_a7d9_74a985c08e99.slice/crio-70f1f012825c2a71817efe2cfeb2995fbda4dc8a67cb3b2c18867ccc4d8ae68c WatchSource:0}: Error finding container 70f1f012825c2a71817efe2cfeb2995fbda4dc8a67cb3b2c18867ccc4d8ae68c: Status 404 returned error can't find the container with id 70f1f012825c2a71817efe2cfeb2995fbda4dc8a67cb3b2c18867ccc4d8ae68c Nov 01 06:40:07.813303 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:07.813262 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b760ca-lfph6" event={"ID":"0d238dea-5ba1-4080-a7d9-74a985c08e99","Type":"ContainerStarted","Data":"dff2bf625e6238497fae749d8f223185ff22fb93814bea246ea25f1c198800e5"} Nov 01 06:40:07.813496 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:07.813321 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b760ca-lfph6" event={"ID":"0d238dea-5ba1-4080-a7d9-74a985c08e99","Type":"ContainerStarted","Data":"70f1f012825c2a71817efe2cfeb2995fbda4dc8a67cb3b2c18867ccc4d8ae68c"} Nov 01 06:40:07.931508 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:07.931438 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b760ca-lfph6" podStartSLOduration=1.9314249129999999 podStartE2EDuration="1.931424913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:40:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:40:07.92983545 +0000 UTC m=+3286.584952618" watchObservedRunningTime="2026-11-01 06:40:07.931424913 +0000 UTC m=+3286.586542079" Nov 01 06:40:08.821430 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:08.821394 5040 generic.go:356] "Generic (PLEG): container finished" podID="0d238dea-5ba1-4080-a7d9-74a985c08e99" containerID="dff2bf625e6238497fae749d8f223185ff22fb93814bea246ea25f1c198800e5" exitCode=0 Nov 01 06:40:08.821887 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:08.821513 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b760ca-lfph6" event={"ID":"0d238dea-5ba1-4080-a7d9-74a985c08e99","Type":"ContainerDied","Data":"dff2bf625e6238497fae749d8f223185ff22fb93814bea246ea25f1c198800e5"} Nov 01 06:40:09.973168 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:40:09.973134 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:40:09.980390 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:09.980366 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b760ca-lfph6" Nov 01 06:40:10.029516 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:10.029445 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d238dea-5ba1-4080-a7d9-74a985c08e99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d238dea-5ba1-4080-a7d9-74a985c08e99-maas-api-openshift-ca\") pod \"0d238dea-5ba1-4080-a7d9-74a985c08e99\" (UID: \"0d238dea-5ba1-4080-a7d9-74a985c08e99\") " Nov 01 06:40:10.029703 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:10.029547 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d238dea-5ba1-4080-a7d9-74a985c08e99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d238dea-5ba1-4080-a7d9-74a985c08e99-maas-api-xks-ca\") pod \"0d238dea-5ba1-4080-a7d9-74a985c08e99\" (UID: \"0d238dea-5ba1-4080-a7d9-74a985c08e99\") " Nov 01 06:40:10.029703 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:10.029645 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d238dea-5ba1-4080-a7d9-74a985c08e99-kube-api-access-qp7n7\" (UniqueName: \"kubernetes.io/projected/0d238dea-5ba1-4080-a7d9-74a985c08e99-kube-api-access-qp7n7\") pod \"0d238dea-5ba1-4080-a7d9-74a985c08e99\" (UID: \"0d238dea-5ba1-4080-a7d9-74a985c08e99\") " Nov 01 06:40:10.029916 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:10.029893 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d238dea-5ba1-4080-a7d9-74a985c08e99-maas-api-openshift-ca" pod "0d238dea-5ba1-4080-a7d9-74a985c08e99" (UID: "0d238dea-5ba1-4080-a7d9-74a985c08e99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:40:10.030062 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:10.030045 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d238dea-5ba1-4080-a7d9-74a985c08e99-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:10.031631 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:10.031609 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d238dea-5ba1-4080-a7d9-74a985c08e99-maas-api-xks-ca" pod "0d238dea-5ba1-4080-a7d9-74a985c08e99" (UID: "0d238dea-5ba1-4080-a7d9-74a985c08e99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:40:10.031718 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:10.031697 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d238dea-5ba1-4080-a7d9-74a985c08e99-kube-api-access-qp7n7" pod "0d238dea-5ba1-4080-a7d9-74a985c08e99" (UID: "0d238dea-5ba1-4080-a7d9-74a985c08e99"). InnerVolumeSpecName "kube-api-access-qp7n7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:40:10.131419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:10.131323 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d238dea-5ba1-4080-a7d9-74a985c08e99-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:10.131419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:10.131364 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp7n7\" (UniqueName: \"kubernetes.io/projected/0d238dea-5ba1-4080-a7d9-74a985c08e99-kube-api-access-qp7n7\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:10.834525 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:10.834474 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b760ca-lfph6" event={"ID":"0d238dea-5ba1-4080-a7d9-74a985c08e99","Type":"ContainerDied","Data":"70f1f012825c2a71817efe2cfeb2995fbda4dc8a67cb3b2c18867ccc4d8ae68c"} Nov 01 06:40:10.834525 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:10.834508 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b760ca-lfph6" Nov 01 06:40:10.834742 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:10.834535 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70f1f012825c2a71817efe2cfeb2995fbda4dc8a67cb3b2c18867ccc4d8ae68c" Nov 01 06:40:16.964036 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:16.963998 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3596b654-cm8gq"] Nov 01 06:40:16.964490 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:16.964464 5040 cpu_manager.go:414] "containerMap: removing container" podUID="0d238dea-5ba1-4080-a7d9-74a985c08e99" containerName="revoke-keys" Nov 01 06:40:16.964557 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:16.964492 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d238dea-5ba1-4080-a7d9-74a985c08e99" containerName="revoke-keys" Nov 01 06:40:16.964606 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:16.964597 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d238dea-5ba1-4080-a7d9-74a985c08e99" containerName="revoke-keys" Nov 01 06:40:16.967926 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:16.967910 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3596b654-cm8gq" Nov 01 06:40:17.003264 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:17.003235 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6cd3d93-fb67-468d-a3af-98a4a460fe82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3596b654-cm8gq\" (UID: \"e6cd3d93-fb67-468d-a3af-98a4a460fe82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3596b654-cm8gq" Nov 01 06:40:17.003440 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:17.003420 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6cd3d93-fb67-468d-a3af-98a4a460fe82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3596b654-cm8gq\" (UID: \"e6cd3d93-fb67-468d-a3af-98a4a460fe82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3596b654-cm8gq" Nov 01 06:40:17.003525 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:17.003501 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwq8b\" (UniqueName: \"kubernetes.io/projected/e6cd3d93-fb67-468d-a3af-98a4a460fe82-kube-api-access-qwq8b\") pod \"maas-api-revoke-sub-3596b654-cm8gq\" (UID: \"e6cd3d93-fb67-468d-a3af-98a4a460fe82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3596b654-cm8gq" Nov 01 06:40:17.021364 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:17.021317 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:40:17.071941 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:17.071909 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:40:17.071941 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:17.071924 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:40:17.072197 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:17.071992 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:40:17.075228 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:17.075201 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3596b654-cm8gq"] Nov 01 06:40:17.104666 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:17.104637 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6cd3d93-fb67-468d-a3af-98a4a460fe82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3596b654-cm8gq\" (UID: \"e6cd3d93-fb67-468d-a3af-98a4a460fe82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3596b654-cm8gq" Nov 01 06:40:17.104806 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:17.104706 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwq8b\" (UniqueName: \"kubernetes.io/projected/e6cd3d93-fb67-468d-a3af-98a4a460fe82-kube-api-access-qwq8b\") pod \"maas-api-revoke-sub-3596b654-cm8gq\" (UID: \"e6cd3d93-fb67-468d-a3af-98a4a460fe82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3596b654-cm8gq" Nov 01 06:40:17.104806 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:17.104752 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6cd3d93-fb67-468d-a3af-98a4a460fe82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3596b654-cm8gq\" (UID: \"e6cd3d93-fb67-468d-a3af-98a4a460fe82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3596b654-cm8gq" Nov 01 06:40:17.105461 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:17.105437 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6cd3d93-fb67-468d-a3af-98a4a460fe82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3596b654-cm8gq\" (UID: \"e6cd3d93-fb67-468d-a3af-98a4a460fe82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3596b654-cm8gq" Nov 01 06:40:17.107134 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:17.107107 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6cd3d93-fb67-468d-a3af-98a4a460fe82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3596b654-cm8gq\" (UID: \"e6cd3d93-fb67-468d-a3af-98a4a460fe82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3596b654-cm8gq" Nov 01 06:40:17.166207 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:17.166176 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwq8b\" (UniqueName: \"kubernetes.io/projected/e6cd3d93-fb67-468d-a3af-98a4a460fe82-kube-api-access-qwq8b\") pod \"maas-api-revoke-sub-3596b654-cm8gq\" (UID: \"e6cd3d93-fb67-468d-a3af-98a4a460fe82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3596b654-cm8gq" Nov 01 06:40:17.280863 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:17.280826 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3596b654-cm8gq" Nov 01 06:40:17.659805 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:17.659778 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3596b654-cm8gq"] Nov 01 06:40:17.661269 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:40:17.661240 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6cd3d93_fb67_468d_a3af_98a4a460fe82.slice/crio-534b273fe0ba005cebb1a56a7790b1326f6c92c5b185d6cc54d01a7d606dd3d2 WatchSource:0}: Error finding container 534b273fe0ba005cebb1a56a7790b1326f6c92c5b185d6cc54d01a7d606dd3d2: Status 404 returned error can't find the container with id 534b273fe0ba005cebb1a56a7790b1326f6c92c5b185d6cc54d01a7d606dd3d2 Nov 01 06:40:17.880253 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:17.880212 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3596b654-cm8gq" event={"ID":"e6cd3d93-fb67-468d-a3af-98a4a460fe82","Type":"ContainerStarted","Data":"534b273fe0ba005cebb1a56a7790b1326f6c92c5b185d6cc54d01a7d606dd3d2"} Nov 01 06:40:18.897038 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:18.896922 5040 generic.go:356] "Generic (PLEG): container finished" podID="e6cd3d93-fb67-468d-a3af-98a4a460fe82" containerID="56eb0f0337dfad14bab3ec88f257a22c104667a0a88c03f4abc222e3d66dd2b0" exitCode=0 Nov 01 06:40:18.897038 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:18.896998 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3596b654-cm8gq" event={"ID":"e6cd3d93-fb67-468d-a3af-98a4a460fe82","Type":"ContainerDied","Data":"56eb0f0337dfad14bab3ec88f257a22c104667a0a88c03f4abc222e3d66dd2b0"} Nov 01 06:40:20.059564 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:20.059536 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3596b654-cm8gq" Nov 01 06:40:20.147354 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:20.147315 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6cd3d93-fb67-468d-a3af-98a4a460fe82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6cd3d93-fb67-468d-a3af-98a4a460fe82-maas-api-xks-ca\") pod \"e6cd3d93-fb67-468d-a3af-98a4a460fe82\" (UID: \"e6cd3d93-fb67-468d-a3af-98a4a460fe82\") " Nov 01 06:40:20.147594 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:20.147399 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6cd3d93-fb67-468d-a3af-98a4a460fe82-kube-api-access-qwq8b\" (UniqueName: \"kubernetes.io/projected/e6cd3d93-fb67-468d-a3af-98a4a460fe82-kube-api-access-qwq8b\") pod \"e6cd3d93-fb67-468d-a3af-98a4a460fe82\" (UID: \"e6cd3d93-fb67-468d-a3af-98a4a460fe82\") " Nov 01 06:40:20.147594 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:20.147461 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6cd3d93-fb67-468d-a3af-98a4a460fe82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6cd3d93-fb67-468d-a3af-98a4a460fe82-maas-api-openshift-ca\") pod \"e6cd3d93-fb67-468d-a3af-98a4a460fe82\" (UID: \"e6cd3d93-fb67-468d-a3af-98a4a460fe82\") " Nov 01 06:40:20.147992 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:20.147955 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6cd3d93-fb67-468d-a3af-98a4a460fe82-maas-api-openshift-ca" pod "e6cd3d93-fb67-468d-a3af-98a4a460fe82" (UID: "e6cd3d93-fb67-468d-a3af-98a4a460fe82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:40:20.149616 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:20.149593 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6cd3d93-fb67-468d-a3af-98a4a460fe82-maas-api-xks-ca" pod "e6cd3d93-fb67-468d-a3af-98a4a460fe82" (UID: "e6cd3d93-fb67-468d-a3af-98a4a460fe82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:40:20.149705 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:20.149615 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6cd3d93-fb67-468d-a3af-98a4a460fe82-kube-api-access-qwq8b" pod "e6cd3d93-fb67-468d-a3af-98a4a460fe82" (UID: "e6cd3d93-fb67-468d-a3af-98a4a460fe82"). InnerVolumeSpecName "kube-api-access-qwq8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:40:20.249743 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:20.249712 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6cd3d93-fb67-468d-a3af-98a4a460fe82-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:20.249743 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:20.249741 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwq8b\" (UniqueName: \"kubernetes.io/projected/e6cd3d93-fb67-468d-a3af-98a4a460fe82-kube-api-access-qwq8b\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:20.249954 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:20.249751 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6cd3d93-fb67-468d-a3af-98a4a460fe82-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:20.911640 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:20.911607 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3596b654-cm8gq" event={"ID":"e6cd3d93-fb67-468d-a3af-98a4a460fe82","Type":"ContainerDied","Data":"534b273fe0ba005cebb1a56a7790b1326f6c92c5b185d6cc54d01a7d606dd3d2"} Nov 01 06:40:20.911796 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:20.911645 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3596b654-cm8gq" Nov 01 06:40:20.911796 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:20.911656 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="534b273fe0ba005cebb1a56a7790b1326f6c92c5b185d6cc54d01a7d606dd3d2" Nov 01 06:40:21.427759 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:21.427720 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5b2be7c-vnmkb"] Nov 01 06:40:21.428230 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:21.428200 5040 cpu_manager.go:414] "containerMap: removing container" podUID="e6cd3d93-fb67-468d-a3af-98a4a460fe82" containerName="revoke-keys" Nov 01 06:40:21.428230 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:21.428211 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6cd3d93-fb67-468d-a3af-98a4a460fe82" containerName="revoke-keys" Nov 01 06:40:21.428364 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:21.428298 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6cd3d93-fb67-468d-a3af-98a4a460fe82" containerName="revoke-keys" Nov 01 06:40:21.431840 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:21.431820 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b2be7c-vnmkb" Nov 01 06:40:21.486532 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:21.485474 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:40:21.486532 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:21.485520 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:40:21.486532 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:21.485582 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:40:21.486532 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:21.485634 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:40:21.490119 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:21.490092 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5b2be7c-vnmkb"] Nov 01 06:40:21.563760 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:21.563731 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddgr9\" (UniqueName: \"kubernetes.io/projected/c71b282f-ff6a-4ba8-aec0-2f651770e690-kube-api-access-ddgr9\") pod \"maas-api-revoke-sub-e5b2be7c-vnmkb\" (UID: \"c71b282f-ff6a-4ba8-aec0-2f651770e690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b2be7c-vnmkb" Nov 01 06:40:21.563931 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:21.563797 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c71b282f-ff6a-4ba8-aec0-2f651770e690-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5b2be7c-vnmkb\" (UID: \"c71b282f-ff6a-4ba8-aec0-2f651770e690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b2be7c-vnmkb" Nov 01 06:40:21.563931 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:21.563883 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c71b282f-ff6a-4ba8-aec0-2f651770e690-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5b2be7c-vnmkb\" (UID: \"c71b282f-ff6a-4ba8-aec0-2f651770e690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b2be7c-vnmkb" Nov 01 06:40:21.665288 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:21.665259 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c71b282f-ff6a-4ba8-aec0-2f651770e690-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5b2be7c-vnmkb\" (UID: \"c71b282f-ff6a-4ba8-aec0-2f651770e690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b2be7c-vnmkb" Nov 01 06:40:21.665442 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:21.665294 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c71b282f-ff6a-4ba8-aec0-2f651770e690-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5b2be7c-vnmkb\" (UID: \"c71b282f-ff6a-4ba8-aec0-2f651770e690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b2be7c-vnmkb" Nov 01 06:40:21.665442 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:21.665384 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddgr9\" (UniqueName: \"kubernetes.io/projected/c71b282f-ff6a-4ba8-aec0-2f651770e690-kube-api-access-ddgr9\") pod \"maas-api-revoke-sub-e5b2be7c-vnmkb\" (UID: \"c71b282f-ff6a-4ba8-aec0-2f651770e690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b2be7c-vnmkb" Nov 01 06:40:21.666043 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:21.666024 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c71b282f-ff6a-4ba8-aec0-2f651770e690-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5b2be7c-vnmkb\" (UID: \"c71b282f-ff6a-4ba8-aec0-2f651770e690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b2be7c-vnmkb" Nov 01 06:40:21.667704 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:21.667672 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c71b282f-ff6a-4ba8-aec0-2f651770e690-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5b2be7c-vnmkb\" (UID: \"c71b282f-ff6a-4ba8-aec0-2f651770e690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b2be7c-vnmkb" Nov 01 06:40:21.727124 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:21.727046 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddgr9\" (UniqueName: \"kubernetes.io/projected/c71b282f-ff6a-4ba8-aec0-2f651770e690-kube-api-access-ddgr9\") pod \"maas-api-revoke-sub-e5b2be7c-vnmkb\" (UID: \"c71b282f-ff6a-4ba8-aec0-2f651770e690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b2be7c-vnmkb" Nov 01 06:40:21.744652 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:21.744621 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b2be7c-vnmkb" Nov 01 06:40:22.122249 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:22.122212 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5b2be7c-vnmkb"] Nov 01 06:40:22.123741 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:40:22.123715 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc71b282f_ff6a_4ba8_aec0_2f651770e690.slice/crio-5e0c1b8e54f290eec4978c66362362e05c0bc66ecffd300145191fdd1e0c59a9 WatchSource:0}: Error finding container 5e0c1b8e54f290eec4978c66362362e05c0bc66ecffd300145191fdd1e0c59a9: Status 404 returned error can't find the container with id 5e0c1b8e54f290eec4978c66362362e05c0bc66ecffd300145191fdd1e0c59a9 Nov 01 06:40:22.152172 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:22.152142 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 06:40:22.153516 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:22.153470 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 06:40:22.926543 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:22.926504 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b2be7c-vnmkb" event={"ID":"c71b282f-ff6a-4ba8-aec0-2f651770e690","Type":"ContainerStarted","Data":"42bcfd40bd356317d7b32b00868b4dc8104aa235bf2ccee9b7d5b02b7e3ede4a"} Nov 01 06:40:22.926936 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:22.926568 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b2be7c-vnmkb" event={"ID":"c71b282f-ff6a-4ba8-aec0-2f651770e690","Type":"ContainerStarted","Data":"5e0c1b8e54f290eec4978c66362362e05c0bc66ecffd300145191fdd1e0c59a9"} Nov 01 06:40:23.044787 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:23.044738 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b2be7c-vnmkb" podStartSLOduration=2.04472323 podStartE2EDuration="2.04472323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:40:23.042753629 +0000 UTC m=+3301.697870796" watchObservedRunningTime="2026-11-01 06:40:23.04472323 +0000 UTC m=+3301.699840397" Nov 01 06:40:23.935782 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:23.935700 5040 generic.go:356] "Generic (PLEG): container finished" podID="c71b282f-ff6a-4ba8-aec0-2f651770e690" containerID="42bcfd40bd356317d7b32b00868b4dc8104aa235bf2ccee9b7d5b02b7e3ede4a" exitCode=0 Nov 01 06:40:23.935782 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:23.935768 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b2be7c-vnmkb" event={"ID":"c71b282f-ff6a-4ba8-aec0-2f651770e690","Type":"ContainerDied","Data":"42bcfd40bd356317d7b32b00868b4dc8104aa235bf2ccee9b7d5b02b7e3ede4a"} Nov 01 06:40:25.078831 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:25.078805 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b2be7c-vnmkb" Nov 01 06:40:25.201800 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:25.201716 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c71b282f-ff6a-4ba8-aec0-2f651770e690-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c71b282f-ff6a-4ba8-aec0-2f651770e690-maas-api-openshift-ca\") pod \"c71b282f-ff6a-4ba8-aec0-2f651770e690\" (UID: \"c71b282f-ff6a-4ba8-aec0-2f651770e690\") " Nov 01 06:40:25.201945 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:25.201862 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c71b282f-ff6a-4ba8-aec0-2f651770e690-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c71b282f-ff6a-4ba8-aec0-2f651770e690-maas-api-xks-ca\") pod \"c71b282f-ff6a-4ba8-aec0-2f651770e690\" (UID: \"c71b282f-ff6a-4ba8-aec0-2f651770e690\") " Nov 01 06:40:25.201945 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:25.201914 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c71b282f-ff6a-4ba8-aec0-2f651770e690-kube-api-access-ddgr9\" (UniqueName: \"kubernetes.io/projected/c71b282f-ff6a-4ba8-aec0-2f651770e690-kube-api-access-ddgr9\") pod \"c71b282f-ff6a-4ba8-aec0-2f651770e690\" (UID: \"c71b282f-ff6a-4ba8-aec0-2f651770e690\") " Nov 01 06:40:25.202165 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:25.202133 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c71b282f-ff6a-4ba8-aec0-2f651770e690-maas-api-openshift-ca" pod "c71b282f-ff6a-4ba8-aec0-2f651770e690" (UID: "c71b282f-ff6a-4ba8-aec0-2f651770e690"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:40:25.204921 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:25.202362 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c71b282f-ff6a-4ba8-aec0-2f651770e690-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:25.204921 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:25.204236 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c71b282f-ff6a-4ba8-aec0-2f651770e690-maas-api-xks-ca" pod "c71b282f-ff6a-4ba8-aec0-2f651770e690" (UID: "c71b282f-ff6a-4ba8-aec0-2f651770e690"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:40:25.204921 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:25.204509 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c71b282f-ff6a-4ba8-aec0-2f651770e690-kube-api-access-ddgr9" pod "c71b282f-ff6a-4ba8-aec0-2f651770e690" (UID: "c71b282f-ff6a-4ba8-aec0-2f651770e690"). InnerVolumeSpecName "kube-api-access-ddgr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:40:25.303770 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:25.303731 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c71b282f-ff6a-4ba8-aec0-2f651770e690-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:25.303770 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:25.303767 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddgr9\" (UniqueName: \"kubernetes.io/projected/c71b282f-ff6a-4ba8-aec0-2f651770e690-kube-api-access-ddgr9\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:25.950849 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:25.950758 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b2be7c-vnmkb" Nov 01 06:40:25.950849 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:25.950784 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b2be7c-vnmkb" event={"ID":"c71b282f-ff6a-4ba8-aec0-2f651770e690","Type":"ContainerDied","Data":"5e0c1b8e54f290eec4978c66362362e05c0bc66ecffd300145191fdd1e0c59a9"} Nov 01 06:40:25.950849 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:25.950830 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e0c1b8e54f290eec4978c66362362e05c0bc66ecffd300145191fdd1e0c59a9" Nov 01 06:40:31.070325 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:31.070287 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1989b21-tm5rg"] Nov 01 06:40:31.105629 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:31.105590 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1989b21-tm5rg"] Nov 01 06:40:31.516226 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:31.516185 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afe34fea-a723-41d1-958b-9adc0bf9d617" path="/var/lib/kubelet/pods/afe34fea-a723-41d1-958b-9adc0bf9d617/volumes" Nov 01 06:40:33.212426 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:33.212392 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk"] Nov 01 06:40:33.213712 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:33.213688 5040 cpu_manager.go:414] "containerMap: removing container" podUID="c71b282f-ff6a-4ba8-aec0-2f651770e690" containerName="revoke-keys" Nov 01 06:40:33.213879 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:33.213864 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c71b282f-ff6a-4ba8-aec0-2f651770e690" containerName="revoke-keys" Nov 01 06:40:33.214123 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:33.214110 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="c71b282f-ff6a-4ba8-aec0-2f651770e690" containerName="revoke-keys" Nov 01 06:40:33.234428 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:33.234394 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk" Nov 01 06:40:33.278501 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:33.264255 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk"] Nov 01 06:40:33.307864 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:33.307834 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-c673e5\"/\"kube-root-ca.crt\"" Nov 01 06:40:33.361616 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:33.361585 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-c673e5\"/\"default-dockercfg-vx7sc\"" Nov 01 06:40:33.374539 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:33.361970 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-c673e5\"/\"explicit-model-c673e5-kserve-self-signed-certs\"" Nov 01 06:40:33.375197 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:33.375171 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-c673e5\"/\"openshift-service-ca.crt\"" Nov 01 06:40:33.392269 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:33.391384 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/801ade29-1ce4-43da-8882-e22206d17f2e-tls-certs\") pod \"explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk\" (UID: \"801ade29-1ce4-43da-8882-e22206d17f2e\") " pod="ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk" Nov 01 06:40:33.392269 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:33.391452 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/801ade29-1ce4-43da-8882-e22206d17f2e-dshm\") pod \"explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk\" (UID: \"801ade29-1ce4-43da-8882-e22206d17f2e\") " pod="ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk" Nov 01 06:40:33.392269 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:33.391507 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw8nv\" (UniqueName: \"kubernetes.io/projected/801ade29-1ce4-43da-8882-e22206d17f2e-kube-api-access-tw8nv\") pod \"explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk\" (UID: \"801ade29-1ce4-43da-8882-e22206d17f2e\") " pod="ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk" Nov 01 06:40:33.392269 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:33.391567 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/801ade29-1ce4-43da-8882-e22206d17f2e-model-cache\") pod \"explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk\" (UID: \"801ade29-1ce4-43da-8882-e22206d17f2e\") " pod="ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk" Nov 01 06:40:33.392269 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:33.391629 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/801ade29-1ce4-43da-8882-e22206d17f2e-tmp-dir\") pod \"explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk\" (UID: \"801ade29-1ce4-43da-8882-e22206d17f2e\") " pod="ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk" Nov 01 06:40:33.392269 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:33.391690 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/801ade29-1ce4-43da-8882-e22206d17f2e-home\") pod \"explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk\" (UID: \"801ade29-1ce4-43da-8882-e22206d17f2e\") " pod="ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk" Nov 01 06:40:33.507500 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:33.506994 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/801ade29-1ce4-43da-8882-e22206d17f2e-tls-certs\") pod \"explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk\" (UID: \"801ade29-1ce4-43da-8882-e22206d17f2e\") " pod="ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk" Nov 01 06:40:33.507500 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:33.507062 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/801ade29-1ce4-43da-8882-e22206d17f2e-dshm\") pod \"explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk\" (UID: \"801ade29-1ce4-43da-8882-e22206d17f2e\") " pod="ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk" Nov 01 06:40:33.507500 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:33.507100 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw8nv\" (UniqueName: \"kubernetes.io/projected/801ade29-1ce4-43da-8882-e22206d17f2e-kube-api-access-tw8nv\") pod \"explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk\" (UID: \"801ade29-1ce4-43da-8882-e22206d17f2e\") " pod="ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk" Nov 01 06:40:33.507500 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:33.507159 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/801ade29-1ce4-43da-8882-e22206d17f2e-model-cache\") pod \"explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk\" (UID: \"801ade29-1ce4-43da-8882-e22206d17f2e\") " pod="ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk" Nov 01 06:40:33.507500 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:33.507223 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/801ade29-1ce4-43da-8882-e22206d17f2e-tmp-dir\") pod \"explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk\" (UID: \"801ade29-1ce4-43da-8882-e22206d17f2e\") " pod="ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk" Nov 01 06:40:33.507500 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:33.507287 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/801ade29-1ce4-43da-8882-e22206d17f2e-home\") pod \"explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk\" (UID: \"801ade29-1ce4-43da-8882-e22206d17f2e\") " pod="ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk" Nov 01 06:40:33.507984 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:33.507776 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/801ade29-1ce4-43da-8882-e22206d17f2e-home\") pod \"explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk\" (UID: \"801ade29-1ce4-43da-8882-e22206d17f2e\") " pod="ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk" Nov 01 06:40:33.509383 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:33.508931 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/801ade29-1ce4-43da-8882-e22206d17f2e-model-cache\") pod \"explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk\" (UID: \"801ade29-1ce4-43da-8882-e22206d17f2e\") " pod="ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk" Nov 01 06:40:33.510566 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:33.509602 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/801ade29-1ce4-43da-8882-e22206d17f2e-tmp-dir\") pod \"explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk\" (UID: \"801ade29-1ce4-43da-8882-e22206d17f2e\") " pod="ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk" Nov 01 06:40:33.514267 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:33.511863 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/801ade29-1ce4-43da-8882-e22206d17f2e-dshm\") pod \"explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk\" (UID: \"801ade29-1ce4-43da-8882-e22206d17f2e\") " pod="ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk" Nov 01 06:40:33.523953 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:33.518008 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/801ade29-1ce4-43da-8882-e22206d17f2e-tls-certs\") pod \"explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk\" (UID: \"801ade29-1ce4-43da-8882-e22206d17f2e\") " pod="ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk" Nov 01 06:40:33.581331 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:33.581297 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw8nv\" (UniqueName: \"kubernetes.io/projected/801ade29-1ce4-43da-8882-e22206d17f2e-kube-api-access-tw8nv\") pod \"explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk\" (UID: \"801ade29-1ce4-43da-8882-e22206d17f2e\") " pod="ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk" Nov 01 06:40:33.585761 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:33.585729 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk" Nov 01 06:40:34.021532 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:34.021500 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk"] Nov 01 06:40:34.022236 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:40:34.022209 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod801ade29_1ce4_43da_8882_e22206d17f2e.slice/crio-5f8dd22d58a3bbe2ecf7a174653bcbccb7c0d064dd29eeb6345b83786257d57b WatchSource:0}: Error finding container 5f8dd22d58a3bbe2ecf7a174653bcbccb7c0d064dd29eeb6345b83786257d57b: Status 404 returned error can't find the container with id 5f8dd22d58a3bbe2ecf7a174653bcbccb7c0d064dd29eeb6345b83786257d57b Nov 01 06:40:34.072103 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:34.072067 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk" event={"ID":"801ade29-1ce4-43da-8882-e22206d17f2e","Type":"ContainerStarted","Data":"5f8dd22d58a3bbe2ecf7a174653bcbccb7c0d064dd29eeb6345b83786257d57b"} Nov 01 06:40:35.081229 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:35.081185 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk" event={"ID":"801ade29-1ce4-43da-8882-e22206d17f2e","Type":"ContainerStarted","Data":"2194a21723ff0249dc0935668e5890545ec11f509fa21f46a39077be5c5b8ed6"} Nov 01 06:40:35.203470 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:35.203404 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk" podStartSLOduration=2.203390423 podStartE2EDuration="2.203390423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:40:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:40:35.200632841 +0000 UTC m=+3313.855750008" watchObservedRunningTime="2026-11-01 06:40:35.203390423 +0000 UTC m=+3313.858507589" Nov 01 06:40:35.586249 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:35.586208 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk" Nov 01 06:40:39.973574 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:40:39.973534 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:40:43.586560 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:43.586526 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk" Nov 01 06:40:43.599319 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:43.599292 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk" Nov 01 06:40:44.150325 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:40:44.150292 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk" Nov 01 06:41:05.279133 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:41:05.279093 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk"] Nov 01 06:41:09.973066 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:41:09.973028 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:41:20.300082 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:41:20.300042 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk" podUID="801ade29-1ce4-43da-8882-e22206d17f2e" containerName="main" containerID="cri-o://2194a21723ff0249dc0935668e5890545ec11f509fa21f46a39077be5c5b8ed6" gracePeriod=45 Nov 01 06:41:20.572410 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:41:20.572385 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk" Nov 01 06:41:20.694237 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:41:20.694207 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/801ade29-1ce4-43da-8882-e22206d17f2e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/801ade29-1ce4-43da-8882-e22206d17f2e-dshm\") pod \"801ade29-1ce4-43da-8882-e22206d17f2e\" (UID: \"801ade29-1ce4-43da-8882-e22206d17f2e\") " Nov 01 06:41:20.694412 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:41:20.694352 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/801ade29-1ce4-43da-8882-e22206d17f2e-kube-api-access-tw8nv\" (UniqueName: \"kubernetes.io/projected/801ade29-1ce4-43da-8882-e22206d17f2e-kube-api-access-tw8nv\") pod \"801ade29-1ce4-43da-8882-e22206d17f2e\" (UID: \"801ade29-1ce4-43da-8882-e22206d17f2e\") " Nov 01 06:41:20.694412 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:41:20.694383 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/801ade29-1ce4-43da-8882-e22206d17f2e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/801ade29-1ce4-43da-8882-e22206d17f2e-tmp-dir\") pod \"801ade29-1ce4-43da-8882-e22206d17f2e\" (UID: \"801ade29-1ce4-43da-8882-e22206d17f2e\") " Nov 01 06:41:20.694412 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:41:20.694406 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/801ade29-1ce4-43da-8882-e22206d17f2e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/801ade29-1ce4-43da-8882-e22206d17f2e-model-cache\") pod \"801ade29-1ce4-43da-8882-e22206d17f2e\" (UID: \"801ade29-1ce4-43da-8882-e22206d17f2e\") " Nov 01 06:41:20.694598 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:41:20.694457 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/801ade29-1ce4-43da-8882-e22206d17f2e-home\" (UniqueName: \"kubernetes.io/empty-dir/801ade29-1ce4-43da-8882-e22206d17f2e-home\") pod \"801ade29-1ce4-43da-8882-e22206d17f2e\" (UID: \"801ade29-1ce4-43da-8882-e22206d17f2e\") " Nov 01 06:41:20.694598 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:41:20.694507 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/801ade29-1ce4-43da-8882-e22206d17f2e-tls-certs\" (UniqueName: \"kubernetes.io/secret/801ade29-1ce4-43da-8882-e22206d17f2e-tls-certs\") pod \"801ade29-1ce4-43da-8882-e22206d17f2e\" (UID: \"801ade29-1ce4-43da-8882-e22206d17f2e\") " Nov 01 06:41:20.694783 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:41:20.694737 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801ade29-1ce4-43da-8882-e22206d17f2e-tmp-dir" pod "801ade29-1ce4-43da-8882-e22206d17f2e" (UID: "801ade29-1ce4-43da-8882-e22206d17f2e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:41:20.694878 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:41:20.694776 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801ade29-1ce4-43da-8882-e22206d17f2e-home" pod "801ade29-1ce4-43da-8882-e22206d17f2e" (UID: "801ade29-1ce4-43da-8882-e22206d17f2e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:41:20.694878 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:41:20.694761 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801ade29-1ce4-43da-8882-e22206d17f2e-model-cache" pod "801ade29-1ce4-43da-8882-e22206d17f2e" (UID: "801ade29-1ce4-43da-8882-e22206d17f2e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:41:20.695117 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:41:20.695084 5040 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/801ade29-1ce4-43da-8882-e22206d17f2e-tmp-dir\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:41:20.695117 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:41:20.695111 5040 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/801ade29-1ce4-43da-8882-e22206d17f2e-model-cache\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:41:20.695324 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:41:20.695125 5040 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/801ade29-1ce4-43da-8882-e22206d17f2e-home\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:41:20.696831 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:41:20.696804 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/801ade29-1ce4-43da-8882-e22206d17f2e-tls-certs" pod "801ade29-1ce4-43da-8882-e22206d17f2e" (UID: "801ade29-1ce4-43da-8882-e22206d17f2e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:41:20.696831 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:41:20.696812 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801ade29-1ce4-43da-8882-e22206d17f2e-dshm" pod "801ade29-1ce4-43da-8882-e22206d17f2e" (UID: "801ade29-1ce4-43da-8882-e22206d17f2e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:41:20.697016 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:41:20.696973 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/801ade29-1ce4-43da-8882-e22206d17f2e-kube-api-access-tw8nv" pod "801ade29-1ce4-43da-8882-e22206d17f2e" (UID: "801ade29-1ce4-43da-8882-e22206d17f2e"). InnerVolumeSpecName "kube-api-access-tw8nv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:41:20.796077 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:41:20.796043 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw8nv\" (UniqueName: \"kubernetes.io/projected/801ade29-1ce4-43da-8882-e22206d17f2e-kube-api-access-tw8nv\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:41:20.796077 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:41:20.796075 5040 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/801ade29-1ce4-43da-8882-e22206d17f2e-tls-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:41:20.796077 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:41:20.796085 5040 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/801ade29-1ce4-43da-8882-e22206d17f2e-dshm\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:41:21.419791 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:41:21.419753 5040 generic.go:356] "Generic (PLEG): container finished" podID="801ade29-1ce4-43da-8882-e22206d17f2e" containerID="2194a21723ff0249dc0935668e5890545ec11f509fa21f46a39077be5c5b8ed6" exitCode=0 Nov 01 06:41:21.420242 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:41:21.419801 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk" Nov 01 06:41:21.420242 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:41:21.419842 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk" event={"ID":"801ade29-1ce4-43da-8882-e22206d17f2e","Type":"ContainerDied","Data":"2194a21723ff0249dc0935668e5890545ec11f509fa21f46a39077be5c5b8ed6"} Nov 01 06:41:21.420242 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:41:21.419883 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk" event={"ID":"801ade29-1ce4-43da-8882-e22206d17f2e","Type":"ContainerDied","Data":"5f8dd22d58a3bbe2ecf7a174653bcbccb7c0d064dd29eeb6345b83786257d57b"} Nov 01 06:41:21.420242 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:41:21.419905 5040 scope.go:122] "RemoveContainer" containerID="2194a21723ff0249dc0935668e5890545ec11f509fa21f46a39077be5c5b8ed6" Nov 01 06:41:21.434132 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:41:21.434105 5040 scope.go:122] "RemoveContainer" containerID="2194a21723ff0249dc0935668e5890545ec11f509fa21f46a39077be5c5b8ed6" Nov 01 06:41:21.434554 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:41:21.434516 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2194a21723ff0249dc0935668e5890545ec11f509fa21f46a39077be5c5b8ed6\": container with ID starting with 2194a21723ff0249dc0935668e5890545ec11f509fa21f46a39077be5c5b8ed6 not found: ID does not exist" containerID="2194a21723ff0249dc0935668e5890545ec11f509fa21f46a39077be5c5b8ed6" Nov 01 06:41:21.434672 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:41:21.434556 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2194a21723ff0249dc0935668e5890545ec11f509fa21f46a39077be5c5b8ed6"} err="failed to get container status \"2194a21723ff0249dc0935668e5890545ec11f509fa21f46a39077be5c5b8ed6\": rpc error: code = NotFound desc = could not find container \"2194a21723ff0249dc0935668e5890545ec11f509fa21f46a39077be5c5b8ed6\": container with ID starting with 2194a21723ff0249dc0935668e5890545ec11f509fa21f46a39077be5c5b8ed6 not found: ID does not exist" Nov 01 06:41:21.588525 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:41:21.588469 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk"] Nov 01 06:41:21.636861 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:41:21.636829 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-c673e5/explicit-model-c673e5-kserve-5b9bf68c6f-m9gjk"] Nov 01 06:41:23.516171 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:41:23.516136 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="801ade29-1ce4-43da-8882-e22206d17f2e" path="/var/lib/kubelet/pods/801ade29-1ce4-43da-8882-e22206d17f2e/volumes" Nov 01 06:41:23.534912 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:41:23.534880 5040 scope.go:122] "RemoveContainer" containerID="55c5326605e256cf25308b0010cdbef212aabc95f722f22d379d4b2a1d96e7b0" Nov 01 06:41:39.972228 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:41:39.972150 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:42:03.103351 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:03.103311 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11a9376b-29m7k"] Nov 01 06:42:03.103892 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:03.103841 5040 cpu_manager.go:414] "containerMap: removing container" podUID="801ade29-1ce4-43da-8882-e22206d17f2e" containerName="main" Nov 01 06:42:03.103892 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:03.103855 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="801ade29-1ce4-43da-8882-e22206d17f2e" containerName="main" Nov 01 06:42:03.103973 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:03.103934 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="801ade29-1ce4-43da-8882-e22206d17f2e" containerName="main" Nov 01 06:42:03.107129 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:03.107113 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a9376b-29m7k" Nov 01 06:42:03.164465 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:03.164432 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11a9376b-29m7k"] Nov 01 06:42:03.210947 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:03.210914 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:42:03.210947 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:03.210933 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:42:03.210947 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:03.210942 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:42:03.211241 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:03.211043 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:42:03.268826 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:03.268794 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33c5b46e-9d7c-4c2b-93a2-245ff352afb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11a9376b-29m7k\" (UID: \"33c5b46e-9d7c-4c2b-93a2-245ff352afb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a9376b-29m7k" Nov 01 06:42:03.269005 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:03.268911 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33c5b46e-9d7c-4c2b-93a2-245ff352afb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11a9376b-29m7k\" (UID: \"33c5b46e-9d7c-4c2b-93a2-245ff352afb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a9376b-29m7k" Nov 01 06:42:03.269005 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:03.268964 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9bcx\" (UniqueName: \"kubernetes.io/projected/33c5b46e-9d7c-4c2b-93a2-245ff352afb3-kube-api-access-p9bcx\") pod \"maas-api-revoke-sub-11a9376b-29m7k\" (UID: \"33c5b46e-9d7c-4c2b-93a2-245ff352afb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a9376b-29m7k" Nov 01 06:42:03.370066 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:03.369979 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9bcx\" (UniqueName: \"kubernetes.io/projected/33c5b46e-9d7c-4c2b-93a2-245ff352afb3-kube-api-access-p9bcx\") pod \"maas-api-revoke-sub-11a9376b-29m7k\" (UID: \"33c5b46e-9d7c-4c2b-93a2-245ff352afb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a9376b-29m7k" Nov 01 06:42:03.370240 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:03.370082 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33c5b46e-9d7c-4c2b-93a2-245ff352afb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11a9376b-29m7k\" (UID: \"33c5b46e-9d7c-4c2b-93a2-245ff352afb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a9376b-29m7k" Nov 01 06:42:03.370240 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:03.370160 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33c5b46e-9d7c-4c2b-93a2-245ff352afb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11a9376b-29m7k\" (UID: \"33c5b46e-9d7c-4c2b-93a2-245ff352afb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a9376b-29m7k" Nov 01 06:42:03.370799 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:03.370778 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33c5b46e-9d7c-4c2b-93a2-245ff352afb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11a9376b-29m7k\" (UID: \"33c5b46e-9d7c-4c2b-93a2-245ff352afb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a9376b-29m7k" Nov 01 06:42:03.372527 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:03.372504 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33c5b46e-9d7c-4c2b-93a2-245ff352afb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11a9376b-29m7k\" (UID: \"33c5b46e-9d7c-4c2b-93a2-245ff352afb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a9376b-29m7k" Nov 01 06:42:03.431875 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:03.431841 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9bcx\" (UniqueName: \"kubernetes.io/projected/33c5b46e-9d7c-4c2b-93a2-245ff352afb3-kube-api-access-p9bcx\") pod \"maas-api-revoke-sub-11a9376b-29m7k\" (UID: \"33c5b46e-9d7c-4c2b-93a2-245ff352afb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a9376b-29m7k" Nov 01 06:42:03.720850 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:03.720767 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a9376b-29m7k" Nov 01 06:42:04.098156 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:04.098126 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11a9376b-29m7k"] Nov 01 06:42:04.099302 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:42:04.099277 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33c5b46e_9d7c_4c2b_93a2_245ff352afb3.slice/crio-ad06ef75ea2c4050c0d807db3f1485f4b7dba5b00c8cddd1b63cb2123a116623 WatchSource:0}: Error finding container ad06ef75ea2c4050c0d807db3f1485f4b7dba5b00c8cddd1b63cb2123a116623: Status 404 returned error can't find the container with id ad06ef75ea2c4050c0d807db3f1485f4b7dba5b00c8cddd1b63cb2123a116623 Nov 01 06:42:04.727742 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:04.727638 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a9376b-29m7k" event={"ID":"33c5b46e-9d7c-4c2b-93a2-245ff352afb3","Type":"ContainerStarted","Data":"8a3e1ae9ca705270509db3fbcf635c70ff4e4e776d83e77beca7d342560dbb6b"} Nov 01 06:42:04.727742 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:04.727735 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a9376b-29m7k" event={"ID":"33c5b46e-9d7c-4c2b-93a2-245ff352afb3","Type":"ContainerStarted","Data":"ad06ef75ea2c4050c0d807db3f1485f4b7dba5b00c8cddd1b63cb2123a116623"} Nov 01 06:42:04.846279 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:04.846219 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a9376b-29m7k" podStartSLOduration=1.8462052899999999 podStartE2EDuration="1.84620529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:42:04.843891343 +0000 UTC m=+3403.499008509" watchObservedRunningTime="2026-11-01 06:42:04.84620529 +0000 UTC m=+3403.501322457" Nov 01 06:42:05.735822 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:05.735784 5040 generic.go:356] "Generic (PLEG): container finished" podID="33c5b46e-9d7c-4c2b-93a2-245ff352afb3" containerID="8a3e1ae9ca705270509db3fbcf635c70ff4e4e776d83e77beca7d342560dbb6b" exitCode=0 Nov 01 06:42:05.736247 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:05.735868 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a9376b-29m7k" event={"ID":"33c5b46e-9d7c-4c2b-93a2-245ff352afb3","Type":"ContainerDied","Data":"8a3e1ae9ca705270509db3fbcf635c70ff4e4e776d83e77beca7d342560dbb6b"} Nov 01 06:42:06.893836 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:06.893811 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a9376b-29m7k" Nov 01 06:42:07.012390 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:07.012353 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33c5b46e-9d7c-4c2b-93a2-245ff352afb3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33c5b46e-9d7c-4c2b-93a2-245ff352afb3-maas-api-openshift-ca\") pod \"33c5b46e-9d7c-4c2b-93a2-245ff352afb3\" (UID: \"33c5b46e-9d7c-4c2b-93a2-245ff352afb3\") " Nov 01 06:42:07.012656 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:07.012434 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33c5b46e-9d7c-4c2b-93a2-245ff352afb3-kube-api-access-p9bcx\" (UniqueName: \"kubernetes.io/projected/33c5b46e-9d7c-4c2b-93a2-245ff352afb3-kube-api-access-p9bcx\") pod \"33c5b46e-9d7c-4c2b-93a2-245ff352afb3\" (UID: \"33c5b46e-9d7c-4c2b-93a2-245ff352afb3\") " Nov 01 06:42:07.012656 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:07.012534 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33c5b46e-9d7c-4c2b-93a2-245ff352afb3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33c5b46e-9d7c-4c2b-93a2-245ff352afb3-maas-api-xks-ca\") pod \"33c5b46e-9d7c-4c2b-93a2-245ff352afb3\" (UID: \"33c5b46e-9d7c-4c2b-93a2-245ff352afb3\") " Nov 01 06:42:07.012814 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:07.012778 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33c5b46e-9d7c-4c2b-93a2-245ff352afb3-maas-api-openshift-ca" pod "33c5b46e-9d7c-4c2b-93a2-245ff352afb3" (UID: "33c5b46e-9d7c-4c2b-93a2-245ff352afb3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:42:07.013048 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:07.013013 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33c5b46e-9d7c-4c2b-93a2-245ff352afb3-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:42:07.015189 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:07.015154 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33c5b46e-9d7c-4c2b-93a2-245ff352afb3-maas-api-xks-ca" pod "33c5b46e-9d7c-4c2b-93a2-245ff352afb3" (UID: "33c5b46e-9d7c-4c2b-93a2-245ff352afb3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:42:07.015762 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:07.015736 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33c5b46e-9d7c-4c2b-93a2-245ff352afb3-kube-api-access-p9bcx" pod "33c5b46e-9d7c-4c2b-93a2-245ff352afb3" (UID: "33c5b46e-9d7c-4c2b-93a2-245ff352afb3"). InnerVolumeSpecName "kube-api-access-p9bcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:42:07.114530 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:07.114461 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9bcx\" (UniqueName: \"kubernetes.io/projected/33c5b46e-9d7c-4c2b-93a2-245ff352afb3-kube-api-access-p9bcx\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:42:07.114530 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:07.114524 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33c5b46e-9d7c-4c2b-93a2-245ff352afb3-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:42:07.750141 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:07.750110 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a9376b-29m7k" Nov 01 06:42:07.750326 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:07.750152 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a9376b-29m7k" event={"ID":"33c5b46e-9d7c-4c2b-93a2-245ff352afb3","Type":"ContainerDied","Data":"ad06ef75ea2c4050c0d807db3f1485f4b7dba5b00c8cddd1b63cb2123a116623"} Nov 01 06:42:07.750326 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:07.750206 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad06ef75ea2c4050c0d807db3f1485f4b7dba5b00c8cddd1b63cb2123a116623" Nov 01 06:42:09.972218 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:42:09.972180 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:42:21.517377 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:21.517343 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7000af46-hwtsl"] Nov 01 06:42:21.517838 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:21.517820 5040 cpu_manager.go:414] "containerMap: removing container" podUID="33c5b46e-9d7c-4c2b-93a2-245ff352afb3" containerName="revoke-keys" Nov 01 06:42:21.517883 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:21.517839 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33c5b46e-9d7c-4c2b-93a2-245ff352afb3" containerName="revoke-keys" Nov 01 06:42:21.517952 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:21.517943 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="33c5b46e-9d7c-4c2b-93a2-245ff352afb3" containerName="revoke-keys" Nov 01 06:42:21.521225 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:21.521204 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7000af46-hwtsl" Nov 01 06:42:21.572241 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:21.572202 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4jdv\" (UniqueName: \"kubernetes.io/projected/3931ab4a-19a9-48b2-af00-ed8076b78d92-kube-api-access-h4jdv\") pod \"maas-api-revoke-sub-7000af46-hwtsl\" (UID: \"3931ab4a-19a9-48b2-af00-ed8076b78d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7000af46-hwtsl" Nov 01 06:42:21.572451 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:21.572428 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3931ab4a-19a9-48b2-af00-ed8076b78d92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7000af46-hwtsl\" (UID: \"3931ab4a-19a9-48b2-af00-ed8076b78d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7000af46-hwtsl" Nov 01 06:42:21.572569 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:21.572466 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3931ab4a-19a9-48b2-af00-ed8076b78d92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7000af46-hwtsl\" (UID: \"3931ab4a-19a9-48b2-af00-ed8076b78d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7000af46-hwtsl" Nov 01 06:42:21.576373 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:21.576347 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7000af46-hwtsl"] Nov 01 06:42:21.622358 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:21.622327 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:42:21.622358 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:21.622341 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:42:21.622601 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:21.622392 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:42:21.622601 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:21.622399 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:42:21.674075 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:21.674033 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4jdv\" (UniqueName: \"kubernetes.io/projected/3931ab4a-19a9-48b2-af00-ed8076b78d92-kube-api-access-h4jdv\") pod \"maas-api-revoke-sub-7000af46-hwtsl\" (UID: \"3931ab4a-19a9-48b2-af00-ed8076b78d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7000af46-hwtsl" Nov 01 06:42:21.674318 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:21.674150 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3931ab4a-19a9-48b2-af00-ed8076b78d92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7000af46-hwtsl\" (UID: \"3931ab4a-19a9-48b2-af00-ed8076b78d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7000af46-hwtsl" Nov 01 06:42:21.674318 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:21.674169 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3931ab4a-19a9-48b2-af00-ed8076b78d92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7000af46-hwtsl\" (UID: \"3931ab4a-19a9-48b2-af00-ed8076b78d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7000af46-hwtsl" Nov 01 06:42:21.674981 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:21.674958 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3931ab4a-19a9-48b2-af00-ed8076b78d92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7000af46-hwtsl\" (UID: \"3931ab4a-19a9-48b2-af00-ed8076b78d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7000af46-hwtsl" Nov 01 06:42:21.676593 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:21.676574 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3931ab4a-19a9-48b2-af00-ed8076b78d92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7000af46-hwtsl\" (UID: \"3931ab4a-19a9-48b2-af00-ed8076b78d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7000af46-hwtsl" Nov 01 06:42:21.735694 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:21.735656 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4jdv\" (UniqueName: \"kubernetes.io/projected/3931ab4a-19a9-48b2-af00-ed8076b78d92-kube-api-access-h4jdv\") pod \"maas-api-revoke-sub-7000af46-hwtsl\" (UID: \"3931ab4a-19a9-48b2-af00-ed8076b78d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7000af46-hwtsl" Nov 01 06:42:21.834853 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:21.834749 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7000af46-hwtsl" Nov 01 06:42:22.215683 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:22.215647 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7000af46-hwtsl"] Nov 01 06:42:22.216030 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:42:22.216006 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3931ab4a_19a9_48b2_af00_ed8076b78d92.slice/crio-2128c103b6ba00463304d851ea1991adfd8d18a7044dd3c0e7f2d1f684a6f5d8 WatchSource:0}: Error finding container 2128c103b6ba00463304d851ea1991adfd8d18a7044dd3c0e7f2d1f684a6f5d8: Status 404 returned error can't find the container with id 2128c103b6ba00463304d851ea1991adfd8d18a7044dd3c0e7f2d1f684a6f5d8 Nov 01 06:42:22.843543 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:22.843419 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7000af46-hwtsl" event={"ID":"3931ab4a-19a9-48b2-af00-ed8076b78d92","Type":"ContainerStarted","Data":"6cd0db785c6439ed65ef248006198d8caca0c489ffe5cc0eb616a12e1ca16aed"} Nov 01 06:42:22.843543 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:22.843515 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7000af46-hwtsl" event={"ID":"3931ab4a-19a9-48b2-af00-ed8076b78d92","Type":"ContainerStarted","Data":"2128c103b6ba00463304d851ea1991adfd8d18a7044dd3c0e7f2d1f684a6f5d8"} Nov 01 06:42:22.960325 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:22.960257 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7000af46-hwtsl" podStartSLOduration=1.960241832 podStartE2EDuration="1.960241832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:42:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:42:22.959468267 +0000 UTC m=+3421.614585434" watchObservedRunningTime="2026-11-01 06:42:22.960241832 +0000 UTC m=+3421.615358999" Nov 01 06:42:23.564297 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:23.564263 5040 scope.go:122] "RemoveContainer" containerID="787fa3e25940ef0e4f9292146e27c8a54bec22bd95778e30f069ba8eef2190e3" Nov 01 06:42:23.851057 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:23.850966 5040 generic.go:356] "Generic (PLEG): container finished" podID="3931ab4a-19a9-48b2-af00-ed8076b78d92" containerID="6cd0db785c6439ed65ef248006198d8caca0c489ffe5cc0eb616a12e1ca16aed" exitCode=0 Nov 01 06:42:23.851442 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:23.851052 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7000af46-hwtsl" event={"ID":"3931ab4a-19a9-48b2-af00-ed8076b78d92","Type":"ContainerDied","Data":"6cd0db785c6439ed65ef248006198d8caca0c489ffe5cc0eb616a12e1ca16aed"} Nov 01 06:42:25.011819 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:25.011790 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7000af46-hwtsl" Nov 01 06:42:25.112452 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:25.112366 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3931ab4a-19a9-48b2-af00-ed8076b78d92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3931ab4a-19a9-48b2-af00-ed8076b78d92-maas-api-openshift-ca\") pod \"3931ab4a-19a9-48b2-af00-ed8076b78d92\" (UID: \"3931ab4a-19a9-48b2-af00-ed8076b78d92\") " Nov 01 06:42:25.112452 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:25.112441 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3931ab4a-19a9-48b2-af00-ed8076b78d92-kube-api-access-h4jdv\" (UniqueName: \"kubernetes.io/projected/3931ab4a-19a9-48b2-af00-ed8076b78d92-kube-api-access-h4jdv\") pod \"3931ab4a-19a9-48b2-af00-ed8076b78d92\" (UID: \"3931ab4a-19a9-48b2-af00-ed8076b78d92\") " Nov 01 06:42:25.112734 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:25.112579 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3931ab4a-19a9-48b2-af00-ed8076b78d92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3931ab4a-19a9-48b2-af00-ed8076b78d92-maas-api-xks-ca\") pod \"3931ab4a-19a9-48b2-af00-ed8076b78d92\" (UID: \"3931ab4a-19a9-48b2-af00-ed8076b78d92\") " Nov 01 06:42:25.112798 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:25.112737 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3931ab4a-19a9-48b2-af00-ed8076b78d92-maas-api-openshift-ca" pod "3931ab4a-19a9-48b2-af00-ed8076b78d92" (UID: "3931ab4a-19a9-48b2-af00-ed8076b78d92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:42:25.112976 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:25.112955 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3931ab4a-19a9-48b2-af00-ed8076b78d92-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:42:25.114648 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:25.114625 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3931ab4a-19a9-48b2-af00-ed8076b78d92-maas-api-xks-ca" pod "3931ab4a-19a9-48b2-af00-ed8076b78d92" (UID: "3931ab4a-19a9-48b2-af00-ed8076b78d92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:42:25.114755 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:25.114734 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3931ab4a-19a9-48b2-af00-ed8076b78d92-kube-api-access-h4jdv" pod "3931ab4a-19a9-48b2-af00-ed8076b78d92" (UID: "3931ab4a-19a9-48b2-af00-ed8076b78d92"). InnerVolumeSpecName "kube-api-access-h4jdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:42:25.214234 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:25.214192 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4jdv\" (UniqueName: \"kubernetes.io/projected/3931ab4a-19a9-48b2-af00-ed8076b78d92-kube-api-access-h4jdv\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:42:25.214234 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:25.214230 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3931ab4a-19a9-48b2-af00-ed8076b78d92-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:42:25.865336 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:25.865255 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7000af46-hwtsl" Nov 01 06:42:25.865516 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:25.865252 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7000af46-hwtsl" event={"ID":"3931ab4a-19a9-48b2-af00-ed8076b78d92","Type":"ContainerDied","Data":"2128c103b6ba00463304d851ea1991adfd8d18a7044dd3c0e7f2d1f684a6f5d8"} Nov 01 06:42:25.865516 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:25.865377 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2128c103b6ba00463304d851ea1991adfd8d18a7044dd3c0e7f2d1f684a6f5d8" Nov 01 06:42:28.924582 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:28.924543 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb"] Nov 01 06:42:28.925008 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:28.925000 5040 cpu_manager.go:414] "containerMap: removing container" podUID="3931ab4a-19a9-48b2-af00-ed8076b78d92" containerName="revoke-keys" Nov 01 06:42:28.925052 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:28.925012 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3931ab4a-19a9-48b2-af00-ed8076b78d92" containerName="revoke-keys" Nov 01 06:42:28.925149 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:28.925138 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="3931ab4a-19a9-48b2-af00-ed8076b78d92" containerName="revoke-keys" Nov 01 06:42:28.928649 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:28.928623 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" Nov 01 06:42:28.983755 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:28.983717 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-07e4b5-openshift-default-dockercfg-q5p5m\"" Nov 01 06:42:29.055839 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:29.055793 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8578c847-8665-4136-a240-8d92aba810a3-istio-token\") pod \"e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb\" (UID: \"8578c847-8665-4136-a240-8d92aba810a3\") " pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" Nov 01 06:42:29.058929 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:29.057224 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krfbf\" (UniqueName: \"kubernetes.io/projected/8578c847-8665-4136-a240-8d92aba810a3-kube-api-access-krfbf\") pod \"e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb\" (UID: \"8578c847-8665-4136-a240-8d92aba810a3\") " pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" Nov 01 06:42:29.058929 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:29.057289 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8578c847-8665-4136-a240-8d92aba810a3-istio-envoy\") pod \"e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb\" (UID: \"8578c847-8665-4136-a240-8d92aba810a3\") " pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" Nov 01 06:42:29.058929 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:29.057316 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8578c847-8665-4136-a240-8d92aba810a3-istio-data\") pod \"e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb\" (UID: \"8578c847-8665-4136-a240-8d92aba810a3\") " pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" Nov 01 06:42:29.058929 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:29.057440 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8578c847-8665-4136-a240-8d92aba810a3-istio-podinfo\") pod \"e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb\" (UID: \"8578c847-8665-4136-a240-8d92aba810a3\") " pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" Nov 01 06:42:29.058929 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:29.057530 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8578c847-8665-4136-a240-8d92aba810a3-credential-socket\") pod \"e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb\" (UID: \"8578c847-8665-4136-a240-8d92aba810a3\") " pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" Nov 01 06:42:29.058929 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:29.057561 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8578c847-8665-4136-a240-8d92aba810a3-istiod-ca-cert\") pod \"e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb\" (UID: \"8578c847-8665-4136-a240-8d92aba810a3\") " pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" Nov 01 06:42:29.058929 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:29.057596 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8578c847-8665-4136-a240-8d92aba810a3-workload-socket\") pod \"e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb\" (UID: \"8578c847-8665-4136-a240-8d92aba810a3\") " pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" Nov 01 06:42:29.058929 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:29.057624 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8578c847-8665-4136-a240-8d92aba810a3-workload-certs\") pod \"e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb\" (UID: \"8578c847-8665-4136-a240-8d92aba810a3\") " pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" Nov 01 06:42:29.058929 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:29.057111 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb"] Nov 01 06:42:29.158980 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:29.158941 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krfbf\" (UniqueName: \"kubernetes.io/projected/8578c847-8665-4136-a240-8d92aba810a3-kube-api-access-krfbf\") pod \"e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb\" (UID: \"8578c847-8665-4136-a240-8d92aba810a3\") " pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" Nov 01 06:42:29.159172 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:29.158991 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8578c847-8665-4136-a240-8d92aba810a3-istio-envoy\") pod \"e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb\" (UID: \"8578c847-8665-4136-a240-8d92aba810a3\") " pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" Nov 01 06:42:29.159172 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:29.159008 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8578c847-8665-4136-a240-8d92aba810a3-istio-data\") pod \"e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb\" (UID: \"8578c847-8665-4136-a240-8d92aba810a3\") " pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" Nov 01 06:42:29.159172 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:29.159076 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8578c847-8665-4136-a240-8d92aba810a3-istio-podinfo\") pod \"e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb\" (UID: \"8578c847-8665-4136-a240-8d92aba810a3\") " pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" Nov 01 06:42:29.159368 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:29.159205 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8578c847-8665-4136-a240-8d92aba810a3-credential-socket\") pod \"e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb\" (UID: \"8578c847-8665-4136-a240-8d92aba810a3\") " pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" Nov 01 06:42:29.159368 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:29.159252 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8578c847-8665-4136-a240-8d92aba810a3-istiod-ca-cert\") pod \"e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb\" (UID: \"8578c847-8665-4136-a240-8d92aba810a3\") " pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" Nov 01 06:42:29.159368 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:29.159296 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8578c847-8665-4136-a240-8d92aba810a3-workload-socket\") pod \"e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb\" (UID: \"8578c847-8665-4136-a240-8d92aba810a3\") " pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" Nov 01 06:42:29.159368 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:29.159322 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8578c847-8665-4136-a240-8d92aba810a3-workload-certs\") pod \"e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb\" (UID: \"8578c847-8665-4136-a240-8d92aba810a3\") " pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" Nov 01 06:42:29.159627 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:29.159379 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8578c847-8665-4136-a240-8d92aba810a3-istio-token\") pod \"e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb\" (UID: \"8578c847-8665-4136-a240-8d92aba810a3\") " pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" Nov 01 06:42:29.159627 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:29.159532 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8578c847-8665-4136-a240-8d92aba810a3-istio-data\") pod \"e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb\" (UID: \"8578c847-8665-4136-a240-8d92aba810a3\") " pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" Nov 01 06:42:29.159627 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:29.159599 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8578c847-8665-4136-a240-8d92aba810a3-credential-socket\") pod \"e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb\" (UID: \"8578c847-8665-4136-a240-8d92aba810a3\") " pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" Nov 01 06:42:29.159803 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:29.159707 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8578c847-8665-4136-a240-8d92aba810a3-workload-socket\") pod \"e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb\" (UID: \"8578c847-8665-4136-a240-8d92aba810a3\") " pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" Nov 01 06:42:29.159898 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:29.159876 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8578c847-8665-4136-a240-8d92aba810a3-workload-certs\") pod \"e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb\" (UID: \"8578c847-8665-4136-a240-8d92aba810a3\") " pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" Nov 01 06:42:29.160285 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:29.160263 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8578c847-8665-4136-a240-8d92aba810a3-istiod-ca-cert\") pod \"e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb\" (UID: \"8578c847-8665-4136-a240-8d92aba810a3\") " pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" Nov 01 06:42:29.161681 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:29.161646 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8578c847-8665-4136-a240-8d92aba810a3-istio-envoy\") pod \"e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb\" (UID: \"8578c847-8665-4136-a240-8d92aba810a3\") " pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" Nov 01 06:42:29.161890 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:29.161869 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8578c847-8665-4136-a240-8d92aba810a3-istio-podinfo\") pod \"e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb\" (UID: \"8578c847-8665-4136-a240-8d92aba810a3\") " pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" Nov 01 06:42:29.220672 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:29.220590 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8578c847-8665-4136-a240-8d92aba810a3-istio-token\") pod \"e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb\" (UID: \"8578c847-8665-4136-a240-8d92aba810a3\") " pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" Nov 01 06:42:29.220833 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:29.220717 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krfbf\" (UniqueName: \"kubernetes.io/projected/8578c847-8665-4136-a240-8d92aba810a3-kube-api-access-krfbf\") pod \"e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb\" (UID: \"8578c847-8665-4136-a240-8d92aba810a3\") " pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" Nov 01 06:42:29.242466 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:29.242430 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" Nov 01 06:42:29.684149 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:29.684107 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb"] Nov 01 06:42:29.685084 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:42:29.685053 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8578c847_8665_4136_a240_8d92aba810a3.slice/crio-9338236c72b27dafd72bfd0437795dea953b07b1df2e6110573fb73a146714de WatchSource:0}: Error finding container 9338236c72b27dafd72bfd0437795dea953b07b1df2e6110573fb73a146714de: Status 404 returned error can't find the container with id 9338236c72b27dafd72bfd0437795dea953b07b1df2e6110573fb73a146714de Nov 01 06:42:29.687562 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:29.687529 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:42:29.687650 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:29.687601 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:42:29.687650 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:29.687632 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:42:29.894118 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:29.894078 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" event={"ID":"8578c847-8665-4136-a240-8d92aba810a3","Type":"ContainerStarted","Data":"7096db46484f6fbc707bea41351f3f485645593d2ae3ea0a7b0859fe2489c7c0"} Nov 01 06:42:29.894282 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:29.894140 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" event={"ID":"8578c847-8665-4136-a240-8d92aba810a3","Type":"ContainerStarted","Data":"9338236c72b27dafd72bfd0437795dea953b07b1df2e6110573fb73a146714de"} Nov 01 06:42:30.019195 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:30.019137 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" podStartSLOduration=2.019122049 podStartE2EDuration="2.019122049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:42:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:42:30.015297139 +0000 UTC m=+3428.670414343" watchObservedRunningTime="2026-11-01 06:42:30.019122049 +0000 UTC m=+3428.674239217" Nov 01 06:42:30.253501 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:30.253009 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" Nov 01 06:42:30.259532 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:30.257598 5040 patch_prober.go:29] interesting pod/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Nov 01 06:42:30.259532 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:30.257658 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" podUID="8578c847-8665-4136-a240-8d92aba810a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Nov 01 06:42:31.257517 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:31.255427 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" Nov 01 06:42:31.524562 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:31.523737 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" podUID="adbac784-4ded-4ca3-b985-3e59983fc637" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 06:42:31.597779 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:31.597733 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" podUID="57cf5e07-36de-48cf-b938-138484cea3a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:42:31.957155 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:31.957071 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" Nov 01 06:42:31.975761 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:31.975727 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" Nov 01 06:42:32.054504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:32.052573 5040 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" podUID="adbac784-4ded-4ca3-b985-3e59983fc637" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:42:32.378937 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:32.378900 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e1417ba-fv8rc"] Nov 01 06:42:32.433217 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:32.433177 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e1417ba-fv8rc" Nov 01 06:42:32.438774 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:32.438736 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" podUID="1878b6fb-f8fe-4f4f-8f4e-056c66fc206a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Nov 01 06:42:32.442076 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:32.442046 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e1417ba-fv8rc"] Nov 01 06:42:32.467120 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:32.466159 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/309dfe7b-1534-4ba4-b376-c28cac0e9229-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e1417ba-fv8rc\" (UID: \"309dfe7b-1534-4ba4-b376-c28cac0e9229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e1417ba-fv8rc" Nov 01 06:42:32.467120 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:32.466260 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/309dfe7b-1534-4ba4-b376-c28cac0e9229-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e1417ba-fv8rc\" (UID: \"309dfe7b-1534-4ba4-b376-c28cac0e9229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e1417ba-fv8rc" Nov 01 06:42:32.487498 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:32.487310 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:42:32.487498 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:32.487396 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:42:32.487755 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:32.487520 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:42:32.491507 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:32.490762 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:42:32.568164 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:32.568104 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trspf\" (UniqueName: \"kubernetes.io/projected/309dfe7b-1534-4ba4-b376-c28cac0e9229-kube-api-access-trspf\") pod \"maas-api-revoke-sub-8e1417ba-fv8rc\" (UID: \"309dfe7b-1534-4ba4-b376-c28cac0e9229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e1417ba-fv8rc" Nov 01 06:42:32.568377 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:32.568358 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/309dfe7b-1534-4ba4-b376-c28cac0e9229-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e1417ba-fv8rc\" (UID: \"309dfe7b-1534-4ba4-b376-c28cac0e9229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e1417ba-fv8rc" Nov 01 06:42:32.568492 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:32.568458 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/309dfe7b-1534-4ba4-b376-c28cac0e9229-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e1417ba-fv8rc\" (UID: \"309dfe7b-1534-4ba4-b376-c28cac0e9229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e1417ba-fv8rc" Nov 01 06:42:32.569144 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:32.569121 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/309dfe7b-1534-4ba4-b376-c28cac0e9229-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e1417ba-fv8rc\" (UID: \"309dfe7b-1534-4ba4-b376-c28cac0e9229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e1417ba-fv8rc" Nov 01 06:42:32.570983 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:32.570951 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/309dfe7b-1534-4ba4-b376-c28cac0e9229-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e1417ba-fv8rc\" (UID: \"309dfe7b-1534-4ba4-b376-c28cac0e9229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e1417ba-fv8rc" Nov 01 06:42:32.669548 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:32.669432 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trspf\" (UniqueName: \"kubernetes.io/projected/309dfe7b-1534-4ba4-b376-c28cac0e9229-kube-api-access-trspf\") pod \"maas-api-revoke-sub-8e1417ba-fv8rc\" (UID: \"309dfe7b-1534-4ba4-b376-c28cac0e9229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e1417ba-fv8rc" Nov 01 06:42:32.731160 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:32.731128 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trspf\" (UniqueName: \"kubernetes.io/projected/309dfe7b-1534-4ba4-b376-c28cac0e9229-kube-api-access-trspf\") pod \"maas-api-revoke-sub-8e1417ba-fv8rc\" (UID: \"309dfe7b-1534-4ba4-b376-c28cac0e9229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e1417ba-fv8rc" Nov 01 06:42:32.776274 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:32.776234 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e1417ba-fv8rc" Nov 01 06:42:33.168700 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:33.168670 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e1417ba-fv8rc"] Nov 01 06:42:33.170467 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:42:33.170436 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod309dfe7b_1534_4ba4_b376_c28cac0e9229.slice/crio-70731de8772e3c34a767e1dc13b67b8f45bcb8050bdd265a4ecbc43018dcf275 WatchSource:0}: Error finding container 70731de8772e3c34a767e1dc13b67b8f45bcb8050bdd265a4ecbc43018dcf275: Status 404 returned error can't find the container with id 70731de8772e3c34a767e1dc13b67b8f45bcb8050bdd265a4ecbc43018dcf275 Nov 01 06:42:33.976876 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:33.976770 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e1417ba-fv8rc" event={"ID":"309dfe7b-1534-4ba4-b376-c28cac0e9229","Type":"ContainerStarted","Data":"441f8987dda188003b200ad57f2623a4d5b7e472581a54162d67e4f63925843e"} Nov 01 06:42:33.977337 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:33.976878 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e1417ba-fv8rc" event={"ID":"309dfe7b-1534-4ba4-b376-c28cac0e9229","Type":"ContainerStarted","Data":"70731de8772e3c34a767e1dc13b67b8f45bcb8050bdd265a4ecbc43018dcf275"} Nov 01 06:42:34.095712 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:34.095655 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e1417ba-fv8rc" podStartSLOduration=2.095640735 podStartE2EDuration="2.095640735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:42:34.094646133 +0000 UTC m=+3432.749763300" watchObservedRunningTime="2026-11-01 06:42:34.095640735 +0000 UTC m=+3432.750757944" Nov 01 06:42:34.262788 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:34.262748 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx"] Nov 01 06:42:34.266571 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:34.266550 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx" Nov 01 06:42:34.288078 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:34.288051 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/377b43b2-5a93-47a9-a7be-2a29f3bec365-home\") pod \"noten-model-07e4b5-kserve-75bc8ff597-kjdcx\" (UID: \"377b43b2-5a93-47a9-a7be-2a29f3bec365\") " pod="ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx" Nov 01 06:42:34.288200 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:34.288087 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/377b43b2-5a93-47a9-a7be-2a29f3bec365-tls-certs\") pod \"noten-model-07e4b5-kserve-75bc8ff597-kjdcx\" (UID: \"377b43b2-5a93-47a9-a7be-2a29f3bec365\") " pod="ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx" Nov 01 06:42:34.288200 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:34.288113 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/377b43b2-5a93-47a9-a7be-2a29f3bec365-dshm\") pod \"noten-model-07e4b5-kserve-75bc8ff597-kjdcx\" (UID: \"377b43b2-5a93-47a9-a7be-2a29f3bec365\") " pod="ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx" Nov 01 06:42:34.288306 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:34.288214 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/377b43b2-5a93-47a9-a7be-2a29f3bec365-model-cache\") pod \"noten-model-07e4b5-kserve-75bc8ff597-kjdcx\" (UID: \"377b43b2-5a93-47a9-a7be-2a29f3bec365\") " pod="ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx" Nov 01 06:42:34.288306 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:34.288272 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/377b43b2-5a93-47a9-a7be-2a29f3bec365-tmp-dir\") pod \"noten-model-07e4b5-kserve-75bc8ff597-kjdcx\" (UID: \"377b43b2-5a93-47a9-a7be-2a29f3bec365\") " pod="ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx" Nov 01 06:42:34.288404 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:34.288352 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpkhj\" (UniqueName: \"kubernetes.io/projected/377b43b2-5a93-47a9-a7be-2a29f3bec365-kube-api-access-lpkhj\") pod \"noten-model-07e4b5-kserve-75bc8ff597-kjdcx\" (UID: \"377b43b2-5a93-47a9-a7be-2a29f3bec365\") " pod="ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx" Nov 01 06:42:34.320648 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:34.320612 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-07e4b5\"/\"default-dockercfg-pwv2q\"" Nov 01 06:42:34.321323 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:34.321020 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-07e4b5\"/\"noten-model-07e4b5-kserve-self-signed-certs\"" Nov 01 06:42:34.321323 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:34.321056 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-07e4b5\"/\"kube-root-ca.crt\"" Nov 01 06:42:34.322646 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:34.322619 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-07e4b5\"/\"openshift-service-ca.crt\"" Nov 01 06:42:34.325275 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:34.325255 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx"] Nov 01 06:42:34.389525 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:34.389458 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/377b43b2-5a93-47a9-a7be-2a29f3bec365-home\") pod \"noten-model-07e4b5-kserve-75bc8ff597-kjdcx\" (UID: \"377b43b2-5a93-47a9-a7be-2a29f3bec365\") " pod="ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx" Nov 01 06:42:34.389525 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:34.389518 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/377b43b2-5a93-47a9-a7be-2a29f3bec365-tls-certs\") pod \"noten-model-07e4b5-kserve-75bc8ff597-kjdcx\" (UID: \"377b43b2-5a93-47a9-a7be-2a29f3bec365\") " pod="ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx" Nov 01 06:42:34.389793 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:34.389550 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/377b43b2-5a93-47a9-a7be-2a29f3bec365-dshm\") pod \"noten-model-07e4b5-kserve-75bc8ff597-kjdcx\" (UID: \"377b43b2-5a93-47a9-a7be-2a29f3bec365\") " pod="ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx" Nov 01 06:42:34.389793 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:34.389571 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/377b43b2-5a93-47a9-a7be-2a29f3bec365-model-cache\") pod \"noten-model-07e4b5-kserve-75bc8ff597-kjdcx\" (UID: \"377b43b2-5a93-47a9-a7be-2a29f3bec365\") " pod="ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx" Nov 01 06:42:34.389793 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:34.389617 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/377b43b2-5a93-47a9-a7be-2a29f3bec365-tmp-dir\") pod \"noten-model-07e4b5-kserve-75bc8ff597-kjdcx\" (UID: \"377b43b2-5a93-47a9-a7be-2a29f3bec365\") " pod="ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx" Nov 01 06:42:34.389793 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:34.389658 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpkhj\" (UniqueName: \"kubernetes.io/projected/377b43b2-5a93-47a9-a7be-2a29f3bec365-kube-api-access-lpkhj\") pod \"noten-model-07e4b5-kserve-75bc8ff597-kjdcx\" (UID: \"377b43b2-5a93-47a9-a7be-2a29f3bec365\") " pod="ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx" Nov 01 06:42:34.390012 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:34.389934 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/377b43b2-5a93-47a9-a7be-2a29f3bec365-home\") pod \"noten-model-07e4b5-kserve-75bc8ff597-kjdcx\" (UID: \"377b43b2-5a93-47a9-a7be-2a29f3bec365\") " pod="ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx" Nov 01 06:42:34.390068 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:34.390012 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/377b43b2-5a93-47a9-a7be-2a29f3bec365-model-cache\") pod \"noten-model-07e4b5-kserve-75bc8ff597-kjdcx\" (UID: \"377b43b2-5a93-47a9-a7be-2a29f3bec365\") " pod="ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx" Nov 01 06:42:34.390129 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:34.390089 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/377b43b2-5a93-47a9-a7be-2a29f3bec365-tmp-dir\") pod \"noten-model-07e4b5-kserve-75bc8ff597-kjdcx\" (UID: \"377b43b2-5a93-47a9-a7be-2a29f3bec365\") " pod="ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx" Nov 01 06:42:34.391878 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:34.391850 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/377b43b2-5a93-47a9-a7be-2a29f3bec365-dshm\") pod \"noten-model-07e4b5-kserve-75bc8ff597-kjdcx\" (UID: \"377b43b2-5a93-47a9-a7be-2a29f3bec365\") " pod="ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx" Nov 01 06:42:34.392277 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:34.392258 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/377b43b2-5a93-47a9-a7be-2a29f3bec365-tls-certs\") pod \"noten-model-07e4b5-kserve-75bc8ff597-kjdcx\" (UID: \"377b43b2-5a93-47a9-a7be-2a29f3bec365\") " pod="ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx" Nov 01 06:42:34.451715 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:34.451683 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpkhj\" (UniqueName: \"kubernetes.io/projected/377b43b2-5a93-47a9-a7be-2a29f3bec365-kube-api-access-lpkhj\") pod \"noten-model-07e4b5-kserve-75bc8ff597-kjdcx\" (UID: \"377b43b2-5a93-47a9-a7be-2a29f3bec365\") " pod="ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx" Nov 01 06:42:34.578773 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:34.578685 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx" Nov 01 06:42:34.978427 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:42:34.978388 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod377b43b2_5a93_47a9_a7be_2a29f3bec365.slice/crio-7a4e0d12936d0f704314556d8a5f121026fdeb666ede76ef2a20b8312909e0c3 WatchSource:0}: Error finding container 7a4e0d12936d0f704314556d8a5f121026fdeb666ede76ef2a20b8312909e0c3: Status 404 returned error can't find the container with id 7a4e0d12936d0f704314556d8a5f121026fdeb666ede76ef2a20b8312909e0c3 Nov 01 06:42:34.978876 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:34.978723 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx"] Nov 01 06:42:34.985944 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:34.985916 5040 generic.go:356] "Generic (PLEG): container finished" podID="309dfe7b-1534-4ba4-b376-c28cac0e9229" containerID="441f8987dda188003b200ad57f2623a4d5b7e472581a54162d67e4f63925843e" exitCode=0 Nov 01 06:42:34.986074 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:34.986002 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e1417ba-fv8rc" event={"ID":"309dfe7b-1534-4ba4-b376-c28cac0e9229","Type":"ContainerDied","Data":"441f8987dda188003b200ad57f2623a4d5b7e472581a54162d67e4f63925843e"} Nov 01 06:42:35.994271 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:35.994224 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx" event={"ID":"377b43b2-5a93-47a9-a7be-2a29f3bec365","Type":"ContainerStarted","Data":"d924b17179cbff9c8f1e7cd4194db036b7375ec4dc2640cc00380ed7aee0cf22"} Nov 01 06:42:35.994271 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:35.994278 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx" event={"ID":"377b43b2-5a93-47a9-a7be-2a29f3bec365","Type":"ContainerStarted","Data":"7a4e0d12936d0f704314556d8a5f121026fdeb666ede76ef2a20b8312909e0c3"} Nov 01 06:42:36.114949 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:36.114890 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx" podStartSLOduration=2.114874618 podStartE2EDuration="2.114874618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:42:36.113662894 +0000 UTC m=+3434.768780061" watchObservedRunningTime="2026-11-01 06:42:36.114874618 +0000 UTC m=+3434.769991784" Nov 01 06:42:36.136095 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:36.136070 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e1417ba-fv8rc" Nov 01 06:42:36.208714 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:36.208669 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/309dfe7b-1534-4ba4-b376-c28cac0e9229-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/309dfe7b-1534-4ba4-b376-c28cac0e9229-maas-api-openshift-ca\") pod \"309dfe7b-1534-4ba4-b376-c28cac0e9229\" (UID: \"309dfe7b-1534-4ba4-b376-c28cac0e9229\") " Nov 01 06:42:36.208916 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:36.208891 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/309dfe7b-1534-4ba4-b376-c28cac0e9229-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/309dfe7b-1534-4ba4-b376-c28cac0e9229-maas-api-xks-ca\") pod \"309dfe7b-1534-4ba4-b376-c28cac0e9229\" (UID: \"309dfe7b-1534-4ba4-b376-c28cac0e9229\") " Nov 01 06:42:36.209051 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:36.208968 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/309dfe7b-1534-4ba4-b376-c28cac0e9229-kube-api-access-trspf\" (UniqueName: \"kubernetes.io/projected/309dfe7b-1534-4ba4-b376-c28cac0e9229-kube-api-access-trspf\") pod \"309dfe7b-1534-4ba4-b376-c28cac0e9229\" (UID: \"309dfe7b-1534-4ba4-b376-c28cac0e9229\") " Nov 01 06:42:36.209104 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:36.209062 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/309dfe7b-1534-4ba4-b376-c28cac0e9229-maas-api-openshift-ca" pod "309dfe7b-1534-4ba4-b376-c28cac0e9229" (UID: "309dfe7b-1534-4ba4-b376-c28cac0e9229"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:42:36.209318 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:36.209293 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/309dfe7b-1534-4ba4-b376-c28cac0e9229-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:42:36.210946 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:36.210919 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/309dfe7b-1534-4ba4-b376-c28cac0e9229-maas-api-xks-ca" pod "309dfe7b-1534-4ba4-b376-c28cac0e9229" (UID: "309dfe7b-1534-4ba4-b376-c28cac0e9229"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:42:36.211049 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:36.211027 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/309dfe7b-1534-4ba4-b376-c28cac0e9229-kube-api-access-trspf" pod "309dfe7b-1534-4ba4-b376-c28cac0e9229" (UID: "309dfe7b-1534-4ba4-b376-c28cac0e9229"). InnerVolumeSpecName "kube-api-access-trspf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:42:36.310353 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:36.310324 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/309dfe7b-1534-4ba4-b376-c28cac0e9229-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:42:36.310353 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:36.310352 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trspf\" (UniqueName: \"kubernetes.io/projected/309dfe7b-1534-4ba4-b376-c28cac0e9229-kube-api-access-trspf\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:42:36.579714 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:36.579632 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx" Nov 01 06:42:37.003811 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:37.003779 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e1417ba-fv8rc" Nov 01 06:42:37.004201 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:37.003810 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e1417ba-fv8rc" event={"ID":"309dfe7b-1534-4ba4-b376-c28cac0e9229","Type":"ContainerDied","Data":"70731de8772e3c34a767e1dc13b67b8f45bcb8050bdd265a4ecbc43018dcf275"} Nov 01 06:42:37.004201 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:37.003859 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70731de8772e3c34a767e1dc13b67b8f45bcb8050bdd265a4ecbc43018dcf275" Nov 01 06:42:39.977111 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:42:39.977066 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:42:44.578884 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:44.578844 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx" Nov 01 06:42:44.592051 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:44.592016 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx" Nov 01 06:42:45.065614 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:42:45.065582 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx" Nov 01 06:43:04.264341 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:04.264292 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx"] Nov 01 06:43:04.444231 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:04.444181 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb"] Nov 01 06:43:04.445117 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:04.444471 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" podUID="8578c847-8665-4136-a240-8d92aba810a3" containerName="istio-proxy" containerID="cri-o://7096db46484f6fbc707bea41351f3f485645593d2ae3ea0a7b0859fe2489c7c0" gracePeriod=30 Nov 01 06:43:04.891760 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:04.891724 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2932d07-rwmhq"] Nov 01 06:43:04.892194 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:04.892181 5040 cpu_manager.go:414] "containerMap: removing container" podUID="309dfe7b-1534-4ba4-b376-c28cac0e9229" containerName="revoke-keys" Nov 01 06:43:04.892242 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:04.892195 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="309dfe7b-1534-4ba4-b376-c28cac0e9229" containerName="revoke-keys" Nov 01 06:43:04.892331 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:04.892320 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="309dfe7b-1534-4ba4-b376-c28cac0e9229" containerName="revoke-keys" Nov 01 06:43:04.895835 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:04.895818 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2932d07-rwmhq" Nov 01 06:43:04.949099 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:04.949060 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:43:04.949280 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:04.949221 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:43:04.953125 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:04.953102 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2932d07-rwmhq"] Nov 01 06:43:05.021180 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:05.021146 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:43:05.021180 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:05.021146 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:43:05.062272 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:05.062236 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c0a1651-82be-44b2-93b2-b9130f494cba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2932d07-rwmhq\" (UID: \"4c0a1651-82be-44b2-93b2-b9130f494cba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2932d07-rwmhq" Nov 01 06:43:05.062441 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:05.062326 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2n4p\" (UniqueName: \"kubernetes.io/projected/4c0a1651-82be-44b2-93b2-b9130f494cba-kube-api-access-r2n4p\") pod \"maas-api-revoke-sub-b2932d07-rwmhq\" (UID: \"4c0a1651-82be-44b2-93b2-b9130f494cba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2932d07-rwmhq" Nov 01 06:43:05.062441 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:05.062425 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c0a1651-82be-44b2-93b2-b9130f494cba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2932d07-rwmhq\" (UID: \"4c0a1651-82be-44b2-93b2-b9130f494cba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2932d07-rwmhq" Nov 01 06:43:05.163655 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:05.163572 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c0a1651-82be-44b2-93b2-b9130f494cba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2932d07-rwmhq\" (UID: \"4c0a1651-82be-44b2-93b2-b9130f494cba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2932d07-rwmhq" Nov 01 06:43:05.163655 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:05.163642 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2n4p\" (UniqueName: \"kubernetes.io/projected/4c0a1651-82be-44b2-93b2-b9130f494cba-kube-api-access-r2n4p\") pod \"maas-api-revoke-sub-b2932d07-rwmhq\" (UID: \"4c0a1651-82be-44b2-93b2-b9130f494cba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2932d07-rwmhq" Nov 01 06:43:05.163863 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:05.163716 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c0a1651-82be-44b2-93b2-b9130f494cba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2932d07-rwmhq\" (UID: \"4c0a1651-82be-44b2-93b2-b9130f494cba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2932d07-rwmhq" Nov 01 06:43:05.164256 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:05.164226 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c0a1651-82be-44b2-93b2-b9130f494cba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2932d07-rwmhq\" (UID: \"4c0a1651-82be-44b2-93b2-b9130f494cba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2932d07-rwmhq" Nov 01 06:43:05.166144 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:05.166116 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c0a1651-82be-44b2-93b2-b9130f494cba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2932d07-rwmhq\" (UID: \"4c0a1651-82be-44b2-93b2-b9130f494cba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2932d07-rwmhq" Nov 01 06:43:05.225107 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:05.225074 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2n4p\" (UniqueName: \"kubernetes.io/projected/4c0a1651-82be-44b2-93b2-b9130f494cba-kube-api-access-r2n4p\") pod \"maas-api-revoke-sub-b2932d07-rwmhq\" (UID: \"4c0a1651-82be-44b2-93b2-b9130f494cba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2932d07-rwmhq" Nov 01 06:43:05.509542 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:05.509511 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2932d07-rwmhq" Nov 01 06:43:05.889069 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:05.889031 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2932d07-rwmhq"] Nov 01 06:43:05.889230 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:43:05.889210 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c0a1651_82be_44b2_93b2_b9130f494cba.slice/crio-d4a29523b7facb3cb11459d8b7af9c149ab2d8aebdc2d225cc7f28b5ee25fcf3 WatchSource:0}: Error finding container d4a29523b7facb3cb11459d8b7af9c149ab2d8aebdc2d225cc7f28b5ee25fcf3: Status 404 returned error can't find the container with id d4a29523b7facb3cb11459d8b7af9c149ab2d8aebdc2d225cc7f28b5ee25fcf3 Nov 01 06:43:06.195558 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:06.195517 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2932d07-rwmhq" event={"ID":"4c0a1651-82be-44b2-93b2-b9130f494cba","Type":"ContainerStarted","Data":"d4a29523b7facb3cb11459d8b7af9c149ab2d8aebdc2d225cc7f28b5ee25fcf3"} Nov 01 06:43:07.203723 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:07.203629 5040 generic.go:356] "Generic (PLEG): container finished" podID="4c0a1651-82be-44b2-93b2-b9130f494cba" containerID="90cd631c2cfb34b2e931f4ec3922b7827e6a8abd0d935c8f51e3322589c03c17" exitCode=0 Nov 01 06:43:07.204089 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:07.203719 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2932d07-rwmhq" event={"ID":"4c0a1651-82be-44b2-93b2-b9130f494cba","Type":"ContainerDied","Data":"90cd631c2cfb34b2e931f4ec3922b7827e6a8abd0d935c8f51e3322589c03c17"} Nov 01 06:43:08.365785 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:08.365758 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2932d07-rwmhq" Nov 01 06:43:08.398838 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:08.398807 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c0a1651-82be-44b2-93b2-b9130f494cba-kube-api-access-r2n4p\" (UniqueName: \"kubernetes.io/projected/4c0a1651-82be-44b2-93b2-b9130f494cba-kube-api-access-r2n4p\") pod \"4c0a1651-82be-44b2-93b2-b9130f494cba\" (UID: \"4c0a1651-82be-44b2-93b2-b9130f494cba\") " Nov 01 06:43:08.398994 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:08.398851 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c0a1651-82be-44b2-93b2-b9130f494cba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c0a1651-82be-44b2-93b2-b9130f494cba-maas-api-openshift-ca\") pod \"4c0a1651-82be-44b2-93b2-b9130f494cba\" (UID: \"4c0a1651-82be-44b2-93b2-b9130f494cba\") " Nov 01 06:43:08.398994 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:08.398954 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c0a1651-82be-44b2-93b2-b9130f494cba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c0a1651-82be-44b2-93b2-b9130f494cba-maas-api-xks-ca\") pod \"4c0a1651-82be-44b2-93b2-b9130f494cba\" (UID: \"4c0a1651-82be-44b2-93b2-b9130f494cba\") " Nov 01 06:43:08.399259 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:08.399233 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c0a1651-82be-44b2-93b2-b9130f494cba-maas-api-openshift-ca" pod "4c0a1651-82be-44b2-93b2-b9130f494cba" (UID: "4c0a1651-82be-44b2-93b2-b9130f494cba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:43:08.400931 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:08.400908 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c0a1651-82be-44b2-93b2-b9130f494cba-kube-api-access-r2n4p" pod "4c0a1651-82be-44b2-93b2-b9130f494cba" (UID: "4c0a1651-82be-44b2-93b2-b9130f494cba"). InnerVolumeSpecName "kube-api-access-r2n4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:43:08.401006 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:08.400981 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c0a1651-82be-44b2-93b2-b9130f494cba-maas-api-xks-ca" pod "4c0a1651-82be-44b2-93b2-b9130f494cba" (UID: "4c0a1651-82be-44b2-93b2-b9130f494cba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:43:08.500096 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:08.500065 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c0a1651-82be-44b2-93b2-b9130f494cba-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:08.500096 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:08.500099 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c0a1651-82be-44b2-93b2-b9130f494cba-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:08.500358 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:08.500112 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2n4p\" (UniqueName: \"kubernetes.io/projected/4c0a1651-82be-44b2-93b2-b9130f494cba-kube-api-access-r2n4p\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:09.218505 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:09.218441 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2932d07-rwmhq" Nov 01 06:43:09.218505 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:09.218462 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2932d07-rwmhq" event={"ID":"4c0a1651-82be-44b2-93b2-b9130f494cba","Type":"ContainerDied","Data":"d4a29523b7facb3cb11459d8b7af9c149ab2d8aebdc2d225cc7f28b5ee25fcf3"} Nov 01 06:43:09.218731 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:09.218570 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4a29523b7facb3cb11459d8b7af9c149ab2d8aebdc2d225cc7f28b5ee25fcf3" Nov 01 06:43:09.703710 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:09.703684 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" Nov 01 06:43:09.709798 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:09.709771 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8578c847-8665-4136-a240-8d92aba810a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8578c847-8665-4136-a240-8d92aba810a3-workload-socket\") pod \"8578c847-8665-4136-a240-8d92aba810a3\" (UID: \"8578c847-8665-4136-a240-8d92aba810a3\") " Nov 01 06:43:09.709916 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:09.709836 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8578c847-8665-4136-a240-8d92aba810a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8578c847-8665-4136-a240-8d92aba810a3-workload-certs\") pod \"8578c847-8665-4136-a240-8d92aba810a3\" (UID: \"8578c847-8665-4136-a240-8d92aba810a3\") " Nov 01 06:43:09.709916 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:09.709858 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8578c847-8665-4136-a240-8d92aba810a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8578c847-8665-4136-a240-8d92aba810a3-istio-envoy\") pod \"8578c847-8665-4136-a240-8d92aba810a3\" (UID: \"8578c847-8665-4136-a240-8d92aba810a3\") " Nov 01 06:43:09.709916 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:09.709904 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8578c847-8665-4136-a240-8d92aba810a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8578c847-8665-4136-a240-8d92aba810a3-istio-podinfo\") pod \"8578c847-8665-4136-a240-8d92aba810a3\" (UID: \"8578c847-8665-4136-a240-8d92aba810a3\") " Nov 01 06:43:09.710086 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:09.709926 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8578c847-8665-4136-a240-8d92aba810a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8578c847-8665-4136-a240-8d92aba810a3-istiod-ca-cert\") pod \"8578c847-8665-4136-a240-8d92aba810a3\" (UID: \"8578c847-8665-4136-a240-8d92aba810a3\") " Nov 01 06:43:09.710086 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:09.709994 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8578c847-8665-4136-a240-8d92aba810a3-workload-socket" pod "8578c847-8665-4136-a240-8d92aba810a3" (UID: "8578c847-8665-4136-a240-8d92aba810a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:43:09.710086 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:09.710042 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8578c847-8665-4136-a240-8d92aba810a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8578c847-8665-4136-a240-8d92aba810a3-credential-socket\") pod \"8578c847-8665-4136-a240-8d92aba810a3\" (UID: \"8578c847-8665-4136-a240-8d92aba810a3\") " Nov 01 06:43:09.710254 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:09.710086 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8578c847-8665-4136-a240-8d92aba810a3-kube-api-access-krfbf\" (UniqueName: \"kubernetes.io/projected/8578c847-8665-4136-a240-8d92aba810a3-kube-api-access-krfbf\") pod \"8578c847-8665-4136-a240-8d92aba810a3\" (UID: \"8578c847-8665-4136-a240-8d92aba810a3\") " Nov 01 06:43:09.710254 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:09.710216 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8578c847-8665-4136-a240-8d92aba810a3-workload-certs" pod "8578c847-8665-4136-a240-8d92aba810a3" (UID: "8578c847-8665-4136-a240-8d92aba810a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:43:09.710254 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:09.710201 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8578c847-8665-4136-a240-8d92aba810a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8578c847-8665-4136-a240-8d92aba810a3-istio-data\") pod \"8578c847-8665-4136-a240-8d92aba810a3\" (UID: \"8578c847-8665-4136-a240-8d92aba810a3\") " Nov 01 06:43:09.710433 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:09.710307 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8578c847-8665-4136-a240-8d92aba810a3-credential-socket" pod "8578c847-8665-4136-a240-8d92aba810a3" (UID: "8578c847-8665-4136-a240-8d92aba810a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:43:09.710433 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:09.710347 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8578c847-8665-4136-a240-8d92aba810a3-istiod-ca-cert" pod "8578c847-8665-4136-a240-8d92aba810a3" (UID: "8578c847-8665-4136-a240-8d92aba810a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:43:09.710575 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:09.710472 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8578c847-8665-4136-a240-8d92aba810a3-istio-data" pod "8578c847-8665-4136-a240-8d92aba810a3" (UID: "8578c847-8665-4136-a240-8d92aba810a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:43:09.710788 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:09.710763 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8578c847-8665-4136-a240-8d92aba810a3-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:09.710788 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:09.710789 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8578c847-8665-4136-a240-8d92aba810a3-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:09.710985 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:09.710799 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8578c847-8665-4136-a240-8d92aba810a3-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:09.710985 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:09.710814 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8578c847-8665-4136-a240-8d92aba810a3-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:09.710985 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:09.710830 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8578c847-8665-4136-a240-8d92aba810a3-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:09.712347 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:09.712327 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8578c847-8665-4136-a240-8d92aba810a3-kube-api-access-krfbf" pod "8578c847-8665-4136-a240-8d92aba810a3" (UID: "8578c847-8665-4136-a240-8d92aba810a3"). InnerVolumeSpecName "kube-api-access-krfbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:43:09.712594 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:09.712571 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8578c847-8665-4136-a240-8d92aba810a3-istio-envoy" pod "8578c847-8665-4136-a240-8d92aba810a3" (UID: "8578c847-8665-4136-a240-8d92aba810a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:43:09.712688 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:09.712614 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8578c847-8665-4136-a240-8d92aba810a3-istio-podinfo" pod "8578c847-8665-4136-a240-8d92aba810a3" (UID: "8578c847-8665-4136-a240-8d92aba810a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:43:09.811787 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:09.811753 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8578c847-8665-4136-a240-8d92aba810a3-istio-token\" (UniqueName: \"kubernetes.io/projected/8578c847-8665-4136-a240-8d92aba810a3-istio-token\") pod \"8578c847-8665-4136-a240-8d92aba810a3\" (UID: \"8578c847-8665-4136-a240-8d92aba810a3\") " Nov 01 06:43:09.812108 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:09.812092 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8578c847-8665-4136-a240-8d92aba810a3-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:09.812153 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:09.812111 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krfbf\" (UniqueName: \"kubernetes.io/projected/8578c847-8665-4136-a240-8d92aba810a3-kube-api-access-krfbf\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:09.812153 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:09.812121 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8578c847-8665-4136-a240-8d92aba810a3-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:09.813882 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:09.813860 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8578c847-8665-4136-a240-8d92aba810a3-istio-token" pod "8578c847-8665-4136-a240-8d92aba810a3" (UID: "8578c847-8665-4136-a240-8d92aba810a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:43:09.912863 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:09.912826 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8578c847-8665-4136-a240-8d92aba810a3-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:09.973003 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:43:09.972916 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:43:10.226098 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:10.226019 5040 generic.go:356] "Generic (PLEG): container finished" podID="8578c847-8665-4136-a240-8d92aba810a3" containerID="7096db46484f6fbc707bea41351f3f485645593d2ae3ea0a7b0859fe2489c7c0" exitCode=0 Nov 01 06:43:10.226098 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:10.226074 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" Nov 01 06:43:10.226283 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:10.226076 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" event={"ID":"8578c847-8665-4136-a240-8d92aba810a3","Type":"ContainerDied","Data":"7096db46484f6fbc707bea41351f3f485645593d2ae3ea0a7b0859fe2489c7c0"} Nov 01 06:43:10.226283 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:10.226199 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb" event={"ID":"8578c847-8665-4136-a240-8d92aba810a3","Type":"ContainerDied","Data":"9338236c72b27dafd72bfd0437795dea953b07b1df2e6110573fb73a146714de"} Nov 01 06:43:10.226283 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:10.226227 5040 scope.go:122] "RemoveContainer" containerID="7096db46484f6fbc707bea41351f3f485645593d2ae3ea0a7b0859fe2489c7c0" Nov 01 06:43:10.237588 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:10.237563 5040 scope.go:122] "RemoveContainer" containerID="7096db46484f6fbc707bea41351f3f485645593d2ae3ea0a7b0859fe2489c7c0" Nov 01 06:43:10.237919 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:43:10.237891 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7096db46484f6fbc707bea41351f3f485645593d2ae3ea0a7b0859fe2489c7c0\": container with ID starting with 7096db46484f6fbc707bea41351f3f485645593d2ae3ea0a7b0859fe2489c7c0 not found: ID does not exist" containerID="7096db46484f6fbc707bea41351f3f485645593d2ae3ea0a7b0859fe2489c7c0" Nov 01 06:43:10.237983 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:10.237929 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7096db46484f6fbc707bea41351f3f485645593d2ae3ea0a7b0859fe2489c7c0"} err="failed to get container status \"7096db46484f6fbc707bea41351f3f485645593d2ae3ea0a7b0859fe2489c7c0\": rpc error: code = NotFound desc = could not find container \"7096db46484f6fbc707bea41351f3f485645593d2ae3ea0a7b0859fe2489c7c0\": container with ID starting with 7096db46484f6fbc707bea41351f3f485645593d2ae3ea0a7b0859fe2489c7c0 not found: ID does not exist" Nov 01 06:43:10.394135 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:10.394099 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb"] Nov 01 06:43:10.445317 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:10.445286 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-07e4b5-openshift-default-67764994c9-8wfwb"] Nov 01 06:43:11.516568 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:11.516534 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8578c847-8665-4136-a240-8d92aba810a3" path="/var/lib/kubelet/pods/8578c847-8665-4136-a240-8d92aba810a3/volumes" Nov 01 06:43:19.286174 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:19.286105 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx" podUID="377b43b2-5a93-47a9-a7be-2a29f3bec365" containerName="main" containerID="cri-o://d924b17179cbff9c8f1e7cd4194db036b7375ec4dc2640cc00380ed7aee0cf22" gracePeriod=45 Nov 01 06:43:19.559646 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:19.559623 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx" Nov 01 06:43:19.611184 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:19.611153 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377b43b2-5a93-47a9-a7be-2a29f3bec365-dshm\" (UniqueName: \"kubernetes.io/empty-dir/377b43b2-5a93-47a9-a7be-2a29f3bec365-dshm\") pod \"377b43b2-5a93-47a9-a7be-2a29f3bec365\" (UID: \"377b43b2-5a93-47a9-a7be-2a29f3bec365\") " Nov 01 06:43:19.611331 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:19.611287 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377b43b2-5a93-47a9-a7be-2a29f3bec365-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/377b43b2-5a93-47a9-a7be-2a29f3bec365-tmp-dir\") pod \"377b43b2-5a93-47a9-a7be-2a29f3bec365\" (UID: \"377b43b2-5a93-47a9-a7be-2a29f3bec365\") " Nov 01 06:43:19.611380 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:19.611369 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/377b43b2-5a93-47a9-a7be-2a29f3bec365-kube-api-access-lpkhj\" (UniqueName: \"kubernetes.io/projected/377b43b2-5a93-47a9-a7be-2a29f3bec365-kube-api-access-lpkhj\") pod \"377b43b2-5a93-47a9-a7be-2a29f3bec365\" (UID: \"377b43b2-5a93-47a9-a7be-2a29f3bec365\") " Nov 01 06:43:19.611447 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:19.611420 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377b43b2-5a93-47a9-a7be-2a29f3bec365-home\" (UniqueName: \"kubernetes.io/empty-dir/377b43b2-5a93-47a9-a7be-2a29f3bec365-home\") pod \"377b43b2-5a93-47a9-a7be-2a29f3bec365\" (UID: \"377b43b2-5a93-47a9-a7be-2a29f3bec365\") " Nov 01 06:43:19.611639 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:19.611460 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/377b43b2-5a93-47a9-a7be-2a29f3bec365-tls-certs\" (UniqueName: \"kubernetes.io/secret/377b43b2-5a93-47a9-a7be-2a29f3bec365-tls-certs\") pod \"377b43b2-5a93-47a9-a7be-2a29f3bec365\" (UID: \"377b43b2-5a93-47a9-a7be-2a29f3bec365\") " Nov 01 06:43:19.611639 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:19.611507 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377b43b2-5a93-47a9-a7be-2a29f3bec365-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/377b43b2-5a93-47a9-a7be-2a29f3bec365-model-cache\") pod \"377b43b2-5a93-47a9-a7be-2a29f3bec365\" (UID: \"377b43b2-5a93-47a9-a7be-2a29f3bec365\") " Nov 01 06:43:19.611639 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:19.611613 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377b43b2-5a93-47a9-a7be-2a29f3bec365-tmp-dir" pod "377b43b2-5a93-47a9-a7be-2a29f3bec365" (UID: "377b43b2-5a93-47a9-a7be-2a29f3bec365"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:43:19.611853 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:19.611792 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377b43b2-5a93-47a9-a7be-2a29f3bec365-home" pod "377b43b2-5a93-47a9-a7be-2a29f3bec365" (UID: "377b43b2-5a93-47a9-a7be-2a29f3bec365"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:43:19.611913 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:19.611874 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377b43b2-5a93-47a9-a7be-2a29f3bec365-model-cache" pod "377b43b2-5a93-47a9-a7be-2a29f3bec365" (UID: "377b43b2-5a93-47a9-a7be-2a29f3bec365"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:43:19.612018 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:19.612001 5040 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/377b43b2-5a93-47a9-a7be-2a29f3bec365-tmp-dir\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:19.612089 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:19.612018 5040 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/377b43b2-5a93-47a9-a7be-2a29f3bec365-home\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:19.612089 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:19.612027 5040 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/377b43b2-5a93-47a9-a7be-2a29f3bec365-model-cache\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:19.613694 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:19.613666 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/377b43b2-5a93-47a9-a7be-2a29f3bec365-tls-certs" pod "377b43b2-5a93-47a9-a7be-2a29f3bec365" (UID: "377b43b2-5a93-47a9-a7be-2a29f3bec365"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:43:19.613795 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:19.613701 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/377b43b2-5a93-47a9-a7be-2a29f3bec365-kube-api-access-lpkhj" pod "377b43b2-5a93-47a9-a7be-2a29f3bec365" (UID: "377b43b2-5a93-47a9-a7be-2a29f3bec365"). InnerVolumeSpecName "kube-api-access-lpkhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:43:19.613795 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:19.613747 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377b43b2-5a93-47a9-a7be-2a29f3bec365-dshm" pod "377b43b2-5a93-47a9-a7be-2a29f3bec365" (UID: "377b43b2-5a93-47a9-a7be-2a29f3bec365"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:43:19.713263 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:19.713215 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpkhj\" (UniqueName: \"kubernetes.io/projected/377b43b2-5a93-47a9-a7be-2a29f3bec365-kube-api-access-lpkhj\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:19.713263 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:19.713265 5040 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/377b43b2-5a93-47a9-a7be-2a29f3bec365-tls-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:19.713500 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:19.713279 5040 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/377b43b2-5a93-47a9-a7be-2a29f3bec365-dshm\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:20.295880 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:20.295842 5040 generic.go:356] "Generic (PLEG): container finished" podID="377b43b2-5a93-47a9-a7be-2a29f3bec365" containerID="d924b17179cbff9c8f1e7cd4194db036b7375ec4dc2640cc00380ed7aee0cf22" exitCode=0 Nov 01 06:43:20.295880 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:20.295882 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx" Nov 01 06:43:20.296418 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:20.295895 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx" event={"ID":"377b43b2-5a93-47a9-a7be-2a29f3bec365","Type":"ContainerDied","Data":"d924b17179cbff9c8f1e7cd4194db036b7375ec4dc2640cc00380ed7aee0cf22"} Nov 01 06:43:20.296418 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:20.295936 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx" event={"ID":"377b43b2-5a93-47a9-a7be-2a29f3bec365","Type":"ContainerDied","Data":"7a4e0d12936d0f704314556d8a5f121026fdeb666ede76ef2a20b8312909e0c3"} Nov 01 06:43:20.296418 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:20.295960 5040 scope.go:122] "RemoveContainer" containerID="d924b17179cbff9c8f1e7cd4194db036b7375ec4dc2640cc00380ed7aee0cf22" Nov 01 06:43:20.306688 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:20.306669 5040 scope.go:122] "RemoveContainer" containerID="d924b17179cbff9c8f1e7cd4194db036b7375ec4dc2640cc00380ed7aee0cf22" Nov 01 06:43:20.306964 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:43:20.306945 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d924b17179cbff9c8f1e7cd4194db036b7375ec4dc2640cc00380ed7aee0cf22\": container with ID starting with d924b17179cbff9c8f1e7cd4194db036b7375ec4dc2640cc00380ed7aee0cf22 not found: ID does not exist" containerID="d924b17179cbff9c8f1e7cd4194db036b7375ec4dc2640cc00380ed7aee0cf22" Nov 01 06:43:20.307039 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:20.306973 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d924b17179cbff9c8f1e7cd4194db036b7375ec4dc2640cc00380ed7aee0cf22"} err="failed to get container status \"d924b17179cbff9c8f1e7cd4194db036b7375ec4dc2640cc00380ed7aee0cf22\": rpc error: code = NotFound desc = could not find container \"d924b17179cbff9c8f1e7cd4194db036b7375ec4dc2640cc00380ed7aee0cf22\": container with ID starting with d924b17179cbff9c8f1e7cd4194db036b7375ec4dc2640cc00380ed7aee0cf22 not found: ID does not exist" Nov 01 06:43:20.463161 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:20.463121 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx"] Nov 01 06:43:20.468836 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:20.468809 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-07e4b5/noten-model-07e4b5-kserve-75bc8ff597-kjdcx"] Nov 01 06:43:21.516821 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:21.516790 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="377b43b2-5a93-47a9-a7be-2a29f3bec365" path="/var/lib/kubelet/pods/377b43b2-5a93-47a9-a7be-2a29f3bec365/volumes" Nov 01 06:43:27.889525 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:27.889467 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b3cdfa-z9swx"] Nov 01 06:43:27.889975 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:27.889962 5040 cpu_manager.go:414] "containerMap: removing container" podUID="8578c847-8665-4136-a240-8d92aba810a3" containerName="istio-proxy" Nov 01 06:43:27.890039 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:27.889976 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8578c847-8665-4136-a240-8d92aba810a3" containerName="istio-proxy" Nov 01 06:43:27.890039 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:27.889989 5040 cpu_manager.go:414] "containerMap: removing container" podUID="4c0a1651-82be-44b2-93b2-b9130f494cba" containerName="revoke-keys" Nov 01 06:43:27.890039 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:27.889995 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c0a1651-82be-44b2-93b2-b9130f494cba" containerName="revoke-keys" Nov 01 06:43:27.890039 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:27.890034 5040 cpu_manager.go:414] "containerMap: removing container" podUID="377b43b2-5a93-47a9-a7be-2a29f3bec365" containerName="main" Nov 01 06:43:27.890162 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:27.890041 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="377b43b2-5a93-47a9-a7be-2a29f3bec365" containerName="main" Nov 01 06:43:27.890162 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:27.890117 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="377b43b2-5a93-47a9-a7be-2a29f3bec365" containerName="main" Nov 01 06:43:27.890162 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:27.890126 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="8578c847-8665-4136-a240-8d92aba810a3" containerName="istio-proxy" Nov 01 06:43:27.890162 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:27.890134 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c0a1651-82be-44b2-93b2-b9130f494cba" containerName="revoke-keys" Nov 01 06:43:27.893336 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:27.893317 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b3cdfa-z9swx" Nov 01 06:43:27.996994 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:27.996960 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:43:27.996994 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:27.996968 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:43:27.996994 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:27.996990 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:43:27.996994 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:27.996990 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:43:28.000702 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:28.000678 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b3cdfa-z9swx"] Nov 01 06:43:28.004611 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:28.004591 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85b3cdfa-z9swx\" (UID: \"c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b3cdfa-z9swx" Nov 01 06:43:28.004717 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:28.004702 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85b3cdfa-z9swx\" (UID: \"c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b3cdfa-z9swx" Nov 01 06:43:28.004762 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:28.004734 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxlff\" (UniqueName: \"kubernetes.io/projected/c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f-kube-api-access-dxlff\") pod \"maas-api-revoke-sub-85b3cdfa-z9swx\" (UID: \"c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b3cdfa-z9swx" Nov 01 06:43:28.106287 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:28.106247 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85b3cdfa-z9swx\" (UID: \"c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b3cdfa-z9swx" Nov 01 06:43:28.106457 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:28.106342 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85b3cdfa-z9swx\" (UID: \"c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b3cdfa-z9swx" Nov 01 06:43:28.106457 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:28.106365 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxlff\" (UniqueName: \"kubernetes.io/projected/c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f-kube-api-access-dxlff\") pod \"maas-api-revoke-sub-85b3cdfa-z9swx\" (UID: \"c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b3cdfa-z9swx" Nov 01 06:43:28.107077 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:28.107047 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85b3cdfa-z9swx\" (UID: \"c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b3cdfa-z9swx" Nov 01 06:43:28.108684 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:28.108655 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85b3cdfa-z9swx\" (UID: \"c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b3cdfa-z9swx" Nov 01 06:43:28.167913 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:28.167850 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxlff\" (UniqueName: \"kubernetes.io/projected/c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f-kube-api-access-dxlff\") pod \"maas-api-revoke-sub-85b3cdfa-z9swx\" (UID: \"c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b3cdfa-z9swx" Nov 01 06:43:28.206219 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:28.206189 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b3cdfa-z9swx" Nov 01 06:43:28.634242 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:28.634208 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b3cdfa-z9swx"] Nov 01 06:43:28.634540 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:43:28.634511 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6a37b7c_e9ef_4112_9a1f_ff8122c18a5f.slice/crio-fc1ec85659a4ea3682168f0c3be65278e371ec8f02784f52520e74703b6d96a9 WatchSource:0}: Error finding container fc1ec85659a4ea3682168f0c3be65278e371ec8f02784f52520e74703b6d96a9: Status 404 returned error can't find the container with id fc1ec85659a4ea3682168f0c3be65278e371ec8f02784f52520e74703b6d96a9 Nov 01 06:43:29.356193 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:29.356154 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b3cdfa-z9swx" event={"ID":"c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f","Type":"ContainerStarted","Data":"949a6d1f21f5079d7114a9e2768b0c51c0a685052b5e7668119242c365915d8b"} Nov 01 06:43:29.356644 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:29.356218 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b3cdfa-z9swx" event={"ID":"c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f","Type":"ContainerStarted","Data":"fc1ec85659a4ea3682168f0c3be65278e371ec8f02784f52520e74703b6d96a9"} Nov 01 06:43:29.474106 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:29.474056 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b3cdfa-z9swx" podStartSLOduration=2.474042438 podStartE2EDuration="2.474042438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:43:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:43:29.472513772 +0000 UTC m=+3488.127630940" watchObservedRunningTime="2026-11-01 06:43:29.474042438 +0000 UTC m=+3488.129159604" Nov 01 06:43:30.364826 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:30.364789 5040 generic.go:356] "Generic (PLEG): container finished" podID="c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f" containerID="949a6d1f21f5079d7114a9e2768b0c51c0a685052b5e7668119242c365915d8b" exitCode=0 Nov 01 06:43:30.365285 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:30.364874 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b3cdfa-z9swx" event={"ID":"c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f","Type":"ContainerDied","Data":"949a6d1f21f5079d7114a9e2768b0c51c0a685052b5e7668119242c365915d8b"} Nov 01 06:43:31.541904 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:31.541878 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b3cdfa-z9swx" Nov 01 06:43:31.643084 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:31.643005 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f-kube-api-access-dxlff\" (UniqueName: \"kubernetes.io/projected/c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f-kube-api-access-dxlff\") pod \"c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f\" (UID: \"c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f\") " Nov 01 06:43:31.643234 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:31.643154 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f-maas-api-openshift-ca\") pod \"c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f\" (UID: \"c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f\") " Nov 01 06:43:31.643280 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:31.643246 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f-maas-api-xks-ca\") pod \"c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f\" (UID: \"c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f\") " Nov 01 06:43:31.643565 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:31.643526 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f-maas-api-openshift-ca" pod "c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f" (UID: "c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:43:31.643790 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:31.643755 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:31.645357 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:31.645328 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f-kube-api-access-dxlff" pod "c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f" (UID: "c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f"). InnerVolumeSpecName "kube-api-access-dxlff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:43:31.645683 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:31.645658 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f-maas-api-xks-ca" pod "c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f" (UID: "c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:43:31.745086 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:31.745049 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:31.745086 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:31.745081 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxlff\" (UniqueName: \"kubernetes.io/projected/c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f-kube-api-access-dxlff\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:32.381377 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:32.381345 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b3cdfa-z9swx" Nov 01 06:43:32.381592 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:32.381381 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b3cdfa-z9swx" event={"ID":"c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f","Type":"ContainerDied","Data":"fc1ec85659a4ea3682168f0c3be65278e371ec8f02784f52520e74703b6d96a9"} Nov 01 06:43:32.381592 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:32.381428 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc1ec85659a4ea3682168f0c3be65278e371ec8f02784f52520e74703b6d96a9" Nov 01 06:43:35.335233 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:35.335196 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp"] Nov 01 06:43:39.976581 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:43:39.976539 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:43:49.056726 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.056687 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-925683cf-97wz6"] Nov 01 06:43:49.057216 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.057200 5040 cpu_manager.go:414] "containerMap: removing container" podUID="c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f" containerName="revoke-keys" Nov 01 06:43:49.057261 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.057216 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f" containerName="revoke-keys" Nov 01 06:43:49.057312 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.057303 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f" containerName="revoke-keys" Nov 01 06:43:49.060912 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.060890 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925683cf-97wz6" Nov 01 06:43:49.119958 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.119914 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-925683cf-97wz6"] Nov 01 06:43:49.164081 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.164035 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:43:49.164272 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.164119 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:43:49.164272 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.164212 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:43:49.165217 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.165194 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8tcd\" (UniqueName: \"kubernetes.io/projected/09e76dfd-4002-462f-85bd-38d81d2b4046-kube-api-access-q8tcd\") pod \"maas-api-revoke-sub-925683cf-97wz6\" (UID: \"09e76dfd-4002-462f-85bd-38d81d2b4046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925683cf-97wz6" Nov 01 06:43:49.165263 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.165255 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09e76dfd-4002-462f-85bd-38d81d2b4046-maas-api-xks-ca\") pod \"maas-api-revoke-sub-925683cf-97wz6\" (UID: \"09e76dfd-4002-462f-85bd-38d81d2b4046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925683cf-97wz6" Nov 01 06:43:49.165392 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.165370 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09e76dfd-4002-462f-85bd-38d81d2b4046-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-925683cf-97wz6\" (UID: \"09e76dfd-4002-462f-85bd-38d81d2b4046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925683cf-97wz6" Nov 01 06:43:49.166819 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.166797 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:43:49.199796 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.199761 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq"] Nov 01 06:43:49.203710 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.203692 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:43:49.334851 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.334759 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-5be4fa-openshift-default-dockercfg-xllcb\"" Nov 01 06:43:49.344711 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.337350 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09e76dfd-4002-462f-85bd-38d81d2b4046-maas-api-xks-ca\") pod \"maas-api-revoke-sub-925683cf-97wz6\" (UID: \"09e76dfd-4002-462f-85bd-38d81d2b4046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925683cf-97wz6" Nov 01 06:43:49.344711 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.337437 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09e76dfd-4002-462f-85bd-38d81d2b4046-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-925683cf-97wz6\" (UID: \"09e76dfd-4002-462f-85bd-38d81d2b4046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925683cf-97wz6" Nov 01 06:43:49.344711 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.337469 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-credential-socket\") pod \"e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq\" (UID: \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\") " pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:43:49.344711 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.338546 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-istio-envoy\") pod \"e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq\" (UID: \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\") " pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:43:49.344711 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.341909 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-istio-data\") pod \"e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq\" (UID: \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\") " pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:43:49.344711 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.341979 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-istio-podinfo\") pod \"e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq\" (UID: \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\") " pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:43:49.344711 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.342035 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-workload-certs\") pod \"e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq\" (UID: \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\") " pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:43:49.344711 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.342113 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-workload-socket\") pod \"e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq\" (UID: \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\") " pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:43:49.344711 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.342156 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8tcd\" (UniqueName: \"kubernetes.io/projected/09e76dfd-4002-462f-85bd-38d81d2b4046-kube-api-access-q8tcd\") pod \"maas-api-revoke-sub-925683cf-97wz6\" (UID: \"09e76dfd-4002-462f-85bd-38d81d2b4046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925683cf-97wz6" Nov 01 06:43:49.344711 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.342183 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-istiod-ca-cert\") pod \"e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq\" (UID: \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\") " pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:43:49.344711 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.342243 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-istio-token\") pod \"e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq\" (UID: \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\") " pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:43:49.344711 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.342269 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmjpg\" (UniqueName: \"kubernetes.io/projected/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-kube-api-access-mmjpg\") pod \"e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq\" (UID: \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\") " pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:43:49.359843 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.359814 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq"] Nov 01 06:43:49.360791 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.360702 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09e76dfd-4002-462f-85bd-38d81d2b4046-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-925683cf-97wz6\" (UID: \"09e76dfd-4002-462f-85bd-38d81d2b4046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925683cf-97wz6" Nov 01 06:43:49.366793 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.365790 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09e76dfd-4002-462f-85bd-38d81d2b4046-maas-api-xks-ca\") pod \"maas-api-revoke-sub-925683cf-97wz6\" (UID: \"09e76dfd-4002-462f-85bd-38d81d2b4046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925683cf-97wz6" Nov 01 06:43:49.409445 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.409414 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8tcd\" (UniqueName: \"kubernetes.io/projected/09e76dfd-4002-462f-85bd-38d81d2b4046-kube-api-access-q8tcd\") pod \"maas-api-revoke-sub-925683cf-97wz6\" (UID: \"09e76dfd-4002-462f-85bd-38d81d2b4046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925683cf-97wz6" Nov 01 06:43:49.443925 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.443893 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-workload-socket\") pod \"e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq\" (UID: \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\") " pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:43:49.444122 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.443932 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-istiod-ca-cert\") pod \"e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq\" (UID: \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\") " pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:43:49.444122 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.443964 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-istio-token\") pod \"e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq\" (UID: \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\") " pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:43:49.444122 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.444082 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmjpg\" (UniqueName: \"kubernetes.io/projected/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-kube-api-access-mmjpg\") pod \"e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq\" (UID: \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\") " pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:43:49.444323 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.444193 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-credential-socket\") pod \"e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq\" (UID: \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\") " pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:43:49.444323 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.444232 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-istio-envoy\") pod \"e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq\" (UID: \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\") " pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:43:49.444449 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.444330 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-istio-data\") pod \"e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq\" (UID: \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\") " pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:43:49.444449 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.444358 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-workload-socket\") pod \"e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq\" (UID: \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\") " pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:43:49.444449 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.444391 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-istio-podinfo\") pod \"e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq\" (UID: \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\") " pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:43:49.444449 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.444443 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-workload-certs\") pod \"e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq\" (UID: \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\") " pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:43:49.444696 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.444561 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-credential-socket\") pod \"e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq\" (UID: \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\") " pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:43:49.444762 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.444735 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-istio-data\") pod \"e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq\" (UID: \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\") " pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:43:49.444841 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.444823 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-istiod-ca-cert\") pod \"e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq\" (UID: \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\") " pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:43:49.444888 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.444836 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-workload-certs\") pod \"e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq\" (UID: \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\") " pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:43:49.446549 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.446525 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-istio-envoy\") pod \"e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq\" (UID: \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\") " pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:43:49.447063 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.447041 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-istio-podinfo\") pod \"e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq\" (UID: \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\") " pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:43:49.510552 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.507042 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-istio-token\") pod \"e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq\" (UID: \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\") " pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:43:49.510552 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.507267 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmjpg\" (UniqueName: \"kubernetes.io/projected/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-kube-api-access-mmjpg\") pod \"e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq\" (UID: \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\") " pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:43:49.517229 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.517197 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:43:49.678614 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.678579 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925683cf-97wz6" Nov 01 06:43:49.958993 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:43:49.958965 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee73e53_f9e2_49c9_a19c_ee33cc2ac68e.slice/crio-757a02f344613430ccf1e3539d1f76ff6d24c3334824c901ebc8e592817eacc4 WatchSource:0}: Error finding container 757a02f344613430ccf1e3539d1f76ff6d24c3334824c901ebc8e592817eacc4: Status 404 returned error can't find the container with id 757a02f344613430ccf1e3539d1f76ff6d24c3334824c901ebc8e592817eacc4 Nov 01 06:43:49.959204 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.959178 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq"] Nov 01 06:43:49.961625 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.961595 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:43:49.961702 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.961654 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:43:49.961702 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:49.961686 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:43:50.058171 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:50.058133 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-925683cf-97wz6"] Nov 01 06:43:50.355854 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:50.355817 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp" podUID="8acef8ba-1ac8-4cfb-b853-bb9a63afe24e" containerName="main" containerID="cri-o://b38b6aacefa521f04d6e8bbbad1cdadef636ed1eed3527d1787fdc2cad049243" gracePeriod=45 Nov 01 06:43:50.517376 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:50.517333 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925683cf-97wz6" event={"ID":"09e76dfd-4002-462f-85bd-38d81d2b4046","Type":"ContainerStarted","Data":"34474f24678aa795d9ca8b0e988943df18e5928f67e2340257a3760266ae1a24"} Nov 01 06:43:50.517555 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:50.517402 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925683cf-97wz6" event={"ID":"09e76dfd-4002-462f-85bd-38d81d2b4046","Type":"ContainerStarted","Data":"6121264498cfb502fbb82916e2c0fd23b6888d0389bb02d5667f16a19f708b7e"} Nov 01 06:43:50.519170 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:50.519134 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" event={"ID":"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e","Type":"ContainerStarted","Data":"1a88b3ba3539e775c2940edeeb343c0a783531829cc0618570eb68cbb26bfd64"} Nov 01 06:43:50.519315 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:50.519191 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" event={"ID":"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e","Type":"ContainerStarted","Data":"757a02f344613430ccf1e3539d1f76ff6d24c3334824c901ebc8e592817eacc4"} Nov 01 06:43:50.520865 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:50.520835 5040 generic.go:356] "Generic (PLEG): container finished" podID="8acef8ba-1ac8-4cfb-b853-bb9a63afe24e" containerID="b38b6aacefa521f04d6e8bbbad1cdadef636ed1eed3527d1787fdc2cad049243" exitCode=0 Nov 01 06:43:50.520983 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:50.520896 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp" event={"ID":"8acef8ba-1ac8-4cfb-b853-bb9a63afe24e","Type":"ContainerDied","Data":"b38b6aacefa521f04d6e8bbbad1cdadef636ed1eed3527d1787fdc2cad049243"} Nov 01 06:43:50.636189 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:50.636141 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925683cf-97wz6" podStartSLOduration=1.636127696 podStartE2EDuration="1.636127696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:43:50.632932555 +0000 UTC m=+3509.288049721" watchObservedRunningTime="2026-11-01 06:43:50.636127696 +0000 UTC m=+3509.291244896" Nov 01 06:43:50.637668 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:50.637639 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp" Nov 01 06:43:50.671437 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:50.671398 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-kube-api-access-qhjzg\" (UniqueName: \"kubernetes.io/projected/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-kube-api-access-qhjzg\") pod \"8acef8ba-1ac8-4cfb-b853-bb9a63afe24e\" (UID: \"8acef8ba-1ac8-4cfb-b853-bb9a63afe24e\") " Nov 01 06:43:50.671674 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:50.671529 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-tmp-dir\") pod \"8acef8ba-1ac8-4cfb-b853-bb9a63afe24e\" (UID: \"8acef8ba-1ac8-4cfb-b853-bb9a63afe24e\") " Nov 01 06:43:50.671674 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:50.671635 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-tls-certs\" (UniqueName: \"kubernetes.io/secret/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-tls-certs\") pod \"8acef8ba-1ac8-4cfb-b853-bb9a63afe24e\" (UID: \"8acef8ba-1ac8-4cfb-b853-bb9a63afe24e\") " Nov 01 06:43:50.671802 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:50.671681 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-model-cache\") pod \"8acef8ba-1ac8-4cfb-b853-bb9a63afe24e\" (UID: \"8acef8ba-1ac8-4cfb-b853-bb9a63afe24e\") " Nov 01 06:43:50.671802 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:50.671730 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-home\" (UniqueName: \"kubernetes.io/empty-dir/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-home\") pod \"8acef8ba-1ac8-4cfb-b853-bb9a63afe24e\" (UID: \"8acef8ba-1ac8-4cfb-b853-bb9a63afe24e\") " Nov 01 06:43:50.671802 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:50.671778 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-dshm\") pod \"8acef8ba-1ac8-4cfb-b853-bb9a63afe24e\" (UID: \"8acef8ba-1ac8-4cfb-b853-bb9a63afe24e\") " Nov 01 06:43:50.671983 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:50.671941 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-model-cache" pod "8acef8ba-1ac8-4cfb-b853-bb9a63afe24e" (UID: "8acef8ba-1ac8-4cfb-b853-bb9a63afe24e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:43:50.672040 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:50.671958 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-home" pod "8acef8ba-1ac8-4cfb-b853-bb9a63afe24e" (UID: "8acef8ba-1ac8-4cfb-b853-bb9a63afe24e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:43:50.672146 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:50.672116 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-tmp-dir" pod "8acef8ba-1ac8-4cfb-b853-bb9a63afe24e" (UID: "8acef8ba-1ac8-4cfb-b853-bb9a63afe24e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:43:50.672263 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:50.672245 5040 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-tmp-dir\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:50.672338 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:50.672267 5040 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-model-cache\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:50.672338 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:50.672282 5040 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-home\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:50.673915 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:50.673887 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-tls-certs" pod "8acef8ba-1ac8-4cfb-b853-bb9a63afe24e" (UID: "8acef8ba-1ac8-4cfb-b853-bb9a63afe24e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:43:50.674028 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:50.673918 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-dshm" pod "8acef8ba-1ac8-4cfb-b853-bb9a63afe24e" (UID: "8acef8ba-1ac8-4cfb-b853-bb9a63afe24e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:43:50.674028 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:50.673927 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-kube-api-access-qhjzg" pod "8acef8ba-1ac8-4cfb-b853-bb9a63afe24e" (UID: "8acef8ba-1ac8-4cfb-b853-bb9a63afe24e"). InnerVolumeSpecName "kube-api-access-qhjzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:43:50.757496 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:50.757437 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" podStartSLOduration=1.757422635 podStartE2EDuration="1.757422635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:43:50.753815478 +0000 UTC m=+3509.408932646" watchObservedRunningTime="2026-11-01 06:43:50.757422635 +0000 UTC m=+3509.412539802" Nov 01 06:43:50.773676 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:50.773644 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhjzg\" (UniqueName: \"kubernetes.io/projected/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-kube-api-access-qhjzg\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:50.773676 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:50.773672 5040 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-tls-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:50.773676 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:50.773683 5040 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e-dshm\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:51.517515 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:51.517471 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:43:51.524715 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:51.524692 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:43:51.530465 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:51.530438 5040 generic.go:356] "Generic (PLEG): container finished" podID="09e76dfd-4002-462f-85bd-38d81d2b4046" containerID="34474f24678aa795d9ca8b0e988943df18e5928f67e2340257a3760266ae1a24" exitCode=0 Nov 01 06:43:51.530615 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:51.530505 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925683cf-97wz6" event={"ID":"09e76dfd-4002-462f-85bd-38d81d2b4046","Type":"ContainerDied","Data":"34474f24678aa795d9ca8b0e988943df18e5928f67e2340257a3760266ae1a24"} Nov 01 06:43:51.532045 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:51.532030 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp" Nov 01 06:43:51.532142 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:51.532058 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp" event={"ID":"8acef8ba-1ac8-4cfb-b853-bb9a63afe24e","Type":"ContainerDied","Data":"08ea117b8cdb50bf07e8c1366450f16ed868f819699dc103a33f9e79cfb0f62a"} Nov 01 06:43:51.532142 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:51.532089 5040 scope.go:122] "RemoveContainer" containerID="b38b6aacefa521f04d6e8bbbad1cdadef636ed1eed3527d1787fdc2cad049243" Nov 01 06:43:51.532344 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:51.532326 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:43:51.533433 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:51.533414 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:43:51.760553 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:51.760519 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp"] Nov 01 06:43:51.764143 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:51.764117 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-747b7b/rate-test-model-747b7b-kserve-d645d897c-p8qvp"] Nov 01 06:43:52.048937 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:52.048850 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2320dfee-rvn6q"] Nov 01 06:43:52.052137 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:52.052109 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2320dfee-rvn6q"] Nov 01 06:43:52.676150 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:52.676124 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925683cf-97wz6" Nov 01 06:43:52.697119 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:52.697090 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09e76dfd-4002-462f-85bd-38d81d2b4046-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09e76dfd-4002-462f-85bd-38d81d2b4046-maas-api-openshift-ca\") pod \"09e76dfd-4002-462f-85bd-38d81d2b4046\" (UID: \"09e76dfd-4002-462f-85bd-38d81d2b4046\") " Nov 01 06:43:52.697256 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:52.697240 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09e76dfd-4002-462f-85bd-38d81d2b4046-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09e76dfd-4002-462f-85bd-38d81d2b4046-maas-api-xks-ca\") pod \"09e76dfd-4002-462f-85bd-38d81d2b4046\" (UID: \"09e76dfd-4002-462f-85bd-38d81d2b4046\") " Nov 01 06:43:52.697357 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:52.697340 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09e76dfd-4002-462f-85bd-38d81d2b4046-kube-api-access-q8tcd\" (UniqueName: \"kubernetes.io/projected/09e76dfd-4002-462f-85bd-38d81d2b4046-kube-api-access-q8tcd\") pod \"09e76dfd-4002-462f-85bd-38d81d2b4046\" (UID: \"09e76dfd-4002-462f-85bd-38d81d2b4046\") " Nov 01 06:43:52.697512 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:52.697491 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09e76dfd-4002-462f-85bd-38d81d2b4046-maas-api-openshift-ca" pod "09e76dfd-4002-462f-85bd-38d81d2b4046" (UID: "09e76dfd-4002-462f-85bd-38d81d2b4046"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:43:52.697751 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:52.697735 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09e76dfd-4002-462f-85bd-38d81d2b4046-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:52.699251 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:52.699225 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09e76dfd-4002-462f-85bd-38d81d2b4046-maas-api-xks-ca" pod "09e76dfd-4002-462f-85bd-38d81d2b4046" (UID: "09e76dfd-4002-462f-85bd-38d81d2b4046"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:43:52.699438 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:52.699422 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09e76dfd-4002-462f-85bd-38d81d2b4046-kube-api-access-q8tcd" pod "09e76dfd-4002-462f-85bd-38d81d2b4046" (UID: "09e76dfd-4002-462f-85bd-38d81d2b4046"). InnerVolumeSpecName "kube-api-access-q8tcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:43:52.799341 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:52.799258 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8tcd\" (UniqueName: \"kubernetes.io/projected/09e76dfd-4002-462f-85bd-38d81d2b4046-kube-api-access-q8tcd\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:52.799341 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:52.799287 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09e76dfd-4002-462f-85bd-38d81d2b4046-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:53.515555 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:53.515521 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5db3853a-2372-4864-9b1c-baa9d60a3fbc" path="/var/lib/kubelet/pods/5db3853a-2372-4864-9b1c-baa9d60a3fbc/volumes" Nov 01 06:43:53.515989 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:53.515976 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8acef8ba-1ac8-4cfb-b853-bb9a63afe24e" path="/var/lib/kubelet/pods/8acef8ba-1ac8-4cfb-b853-bb9a63afe24e/volumes" Nov 01 06:43:53.547897 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:53.547870 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925683cf-97wz6" Nov 01 06:43:53.548105 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:53.547898 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925683cf-97wz6" event={"ID":"09e76dfd-4002-462f-85bd-38d81d2b4046","Type":"ContainerDied","Data":"6121264498cfb502fbb82916e2c0fd23b6888d0389bb02d5667f16a19f708b7e"} Nov 01 06:43:53.548105 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:43:53.547945 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6121264498cfb502fbb82916e2c0fd23b6888d0389bb02d5667f16a19f708b7e" Nov 01 06:44:06.048128 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:06.048092 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5e240d4-z5jww"] Nov 01 06:44:06.098646 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:06.098606 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5e240d4-z5jww"] Nov 01 06:44:07.514496 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:07.514446 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2b7621d-9ec9-47d9-a515-c9d62ca39799" path="/var/lib/kubelet/pods/e2b7621d-9ec9-47d9-a515-c9d62ca39799/volumes" Nov 01 06:44:09.973010 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:44:09.972969 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:44:13.450714 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:13.450676 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd40336b-gpg8j"] Nov 01 06:44:13.451329 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:13.451309 5040 cpu_manager.go:414] "containerMap: removing container" podUID="8acef8ba-1ac8-4cfb-b853-bb9a63afe24e" containerName="main" Nov 01 06:44:13.451375 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:13.451330 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8acef8ba-1ac8-4cfb-b853-bb9a63afe24e" containerName="main" Nov 01 06:44:13.451375 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:13.451356 5040 cpu_manager.go:414] "containerMap: removing container" podUID="09e76dfd-4002-462f-85bd-38d81d2b4046" containerName="revoke-keys" Nov 01 06:44:13.451375 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:13.451361 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09e76dfd-4002-462f-85bd-38d81d2b4046" containerName="revoke-keys" Nov 01 06:44:13.451494 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:13.451459 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="09e76dfd-4002-462f-85bd-38d81d2b4046" containerName="revoke-keys" Nov 01 06:44:13.451494 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:13.451473 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="8acef8ba-1ac8-4cfb-b853-bb9a63afe24e" containerName="main" Nov 01 06:44:13.455285 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:13.455265 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd40336b-gpg8j" Nov 01 06:44:13.518305 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:13.518267 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd40336b-gpg8j"] Nov 01 06:44:13.524763 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:13.524732 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:44:13.524763 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:13.524735 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:44:13.544736 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:13.544701 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97719997-cc0f-45b0-9816-887a62dde1e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd40336b-gpg8j\" (UID: \"97719997-cc0f-45b0-9816-887a62dde1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd40336b-gpg8j" Nov 01 06:44:13.544927 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:13.544757 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97719997-cc0f-45b0-9816-887a62dde1e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd40336b-gpg8j\" (UID: \"97719997-cc0f-45b0-9816-887a62dde1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd40336b-gpg8j" Nov 01 06:44:13.545001 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:13.544924 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9584\" (UniqueName: \"kubernetes.io/projected/97719997-cc0f-45b0-9816-887a62dde1e4-kube-api-access-z9584\") pod \"maas-api-revoke-sub-fd40336b-gpg8j\" (UID: \"97719997-cc0f-45b0-9816-887a62dde1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd40336b-gpg8j" Nov 01 06:44:13.555999 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:13.555969 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:44:13.556123 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:13.556027 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:44:13.645759 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:13.645723 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9584\" (UniqueName: \"kubernetes.io/projected/97719997-cc0f-45b0-9816-887a62dde1e4-kube-api-access-z9584\") pod \"maas-api-revoke-sub-fd40336b-gpg8j\" (UID: \"97719997-cc0f-45b0-9816-887a62dde1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd40336b-gpg8j" Nov 01 06:44:13.645953 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:13.645844 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97719997-cc0f-45b0-9816-887a62dde1e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd40336b-gpg8j\" (UID: \"97719997-cc0f-45b0-9816-887a62dde1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd40336b-gpg8j" Nov 01 06:44:13.645953 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:13.645903 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97719997-cc0f-45b0-9816-887a62dde1e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd40336b-gpg8j\" (UID: \"97719997-cc0f-45b0-9816-887a62dde1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd40336b-gpg8j" Nov 01 06:44:13.646758 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:13.646735 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97719997-cc0f-45b0-9816-887a62dde1e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd40336b-gpg8j\" (UID: \"97719997-cc0f-45b0-9816-887a62dde1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd40336b-gpg8j" Nov 01 06:44:13.648248 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:13.648229 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97719997-cc0f-45b0-9816-887a62dde1e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd40336b-gpg8j\" (UID: \"97719997-cc0f-45b0-9816-887a62dde1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd40336b-gpg8j" Nov 01 06:44:13.714633 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:13.714560 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9584\" (UniqueName: \"kubernetes.io/projected/97719997-cc0f-45b0-9816-887a62dde1e4-kube-api-access-z9584\") pod \"maas-api-revoke-sub-fd40336b-gpg8j\" (UID: \"97719997-cc0f-45b0-9816-887a62dde1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd40336b-gpg8j" Nov 01 06:44:13.769131 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:13.769085 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd40336b-gpg8j" Nov 01 06:44:14.150012 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:14.149980 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd40336b-gpg8j"] Nov 01 06:44:14.150628 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:44:14.150604 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97719997_cc0f_45b0_9816_887a62dde1e4.slice/crio-ccc7bb99499a0f9916ecbc38f580d1bd43018e4d37b91fba53eb636d02210eab WatchSource:0}: Error finding container ccc7bb99499a0f9916ecbc38f580d1bd43018e4d37b91fba53eb636d02210eab: Status 404 returned error can't find the container with id ccc7bb99499a0f9916ecbc38f580d1bd43018e4d37b91fba53eb636d02210eab Nov 01 06:44:14.687431 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:14.687334 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd40336b-gpg8j" event={"ID":"97719997-cc0f-45b0-9816-887a62dde1e4","Type":"ContainerStarted","Data":"c946728d0cee3e23c124f2a70ca2d8d302c1b0d15aca1ac76fc1ac40c2da9d82"} Nov 01 06:44:14.687431 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:14.687388 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd40336b-gpg8j" event={"ID":"97719997-cc0f-45b0-9816-887a62dde1e4","Type":"ContainerStarted","Data":"ccc7bb99499a0f9916ecbc38f580d1bd43018e4d37b91fba53eb636d02210eab"} Nov 01 06:44:14.804289 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:14.804237 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd40336b-gpg8j" podStartSLOduration=1.8042213550000001 podStartE2EDuration="1.804221355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:44:14.803815986 +0000 UTC m=+3533.458933154" watchObservedRunningTime="2026-11-01 06:44:14.804221355 +0000 UTC m=+3533.459338523" Nov 01 06:44:15.695721 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:15.695682 5040 generic.go:356] "Generic (PLEG): container finished" podID="97719997-cc0f-45b0-9816-887a62dde1e4" containerID="c946728d0cee3e23c124f2a70ca2d8d302c1b0d15aca1ac76fc1ac40c2da9d82" exitCode=0 Nov 01 06:44:15.696123 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:15.695746 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd40336b-gpg8j" event={"ID":"97719997-cc0f-45b0-9816-887a62dde1e4","Type":"ContainerDied","Data":"c946728d0cee3e23c124f2a70ca2d8d302c1b0d15aca1ac76fc1ac40c2da9d82"} Nov 01 06:44:16.836114 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:16.836089 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd40336b-gpg8j" Nov 01 06:44:16.982738 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:16.982656 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97719997-cc0f-45b0-9816-887a62dde1e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97719997-cc0f-45b0-9816-887a62dde1e4-maas-api-xks-ca\") pod \"97719997-cc0f-45b0-9816-887a62dde1e4\" (UID: \"97719997-cc0f-45b0-9816-887a62dde1e4\") " Nov 01 06:44:16.982738 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:16.982696 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97719997-cc0f-45b0-9816-887a62dde1e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97719997-cc0f-45b0-9816-887a62dde1e4-maas-api-openshift-ca\") pod \"97719997-cc0f-45b0-9816-887a62dde1e4\" (UID: \"97719997-cc0f-45b0-9816-887a62dde1e4\") " Nov 01 06:44:16.982931 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:16.982854 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97719997-cc0f-45b0-9816-887a62dde1e4-kube-api-access-z9584\" (UniqueName: \"kubernetes.io/projected/97719997-cc0f-45b0-9816-887a62dde1e4-kube-api-access-z9584\") pod \"97719997-cc0f-45b0-9816-887a62dde1e4\" (UID: \"97719997-cc0f-45b0-9816-887a62dde1e4\") " Nov 01 06:44:16.983179 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:16.983151 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97719997-cc0f-45b0-9816-887a62dde1e4-maas-api-openshift-ca" pod "97719997-cc0f-45b0-9816-887a62dde1e4" (UID: "97719997-cc0f-45b0-9816-887a62dde1e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:44:16.983307 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:16.983276 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97719997-cc0f-45b0-9816-887a62dde1e4-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:44:16.985437 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:16.985409 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97719997-cc0f-45b0-9816-887a62dde1e4-maas-api-xks-ca" pod "97719997-cc0f-45b0-9816-887a62dde1e4" (UID: "97719997-cc0f-45b0-9816-887a62dde1e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:44:16.985591 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:16.985436 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97719997-cc0f-45b0-9816-887a62dde1e4-kube-api-access-z9584" pod "97719997-cc0f-45b0-9816-887a62dde1e4" (UID: "97719997-cc0f-45b0-9816-887a62dde1e4"). InnerVolumeSpecName "kube-api-access-z9584". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:44:17.084058 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:17.084023 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9584\" (UniqueName: \"kubernetes.io/projected/97719997-cc0f-45b0-9816-887a62dde1e4-kube-api-access-z9584\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:44:17.084058 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:17.084055 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97719997-cc0f-45b0-9816-887a62dde1e4-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:44:17.712115 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:17.712081 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd40336b-gpg8j" Nov 01 06:44:17.712115 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:17.712103 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd40336b-gpg8j" event={"ID":"97719997-cc0f-45b0-9816-887a62dde1e4","Type":"ContainerDied","Data":"ccc7bb99499a0f9916ecbc38f580d1bd43018e4d37b91fba53eb636d02210eab"} Nov 01 06:44:17.712399 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:17.712178 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccc7bb99499a0f9916ecbc38f580d1bd43018e4d37b91fba53eb636d02210eab" Nov 01 06:44:18.469232 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:18.469198 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v"] Nov 01 06:44:18.469783 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:18.469766 5040 cpu_manager.go:414] "containerMap: removing container" podUID="97719997-cc0f-45b0-9816-887a62dde1e4" containerName="revoke-keys" Nov 01 06:44:18.469826 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:18.469784 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97719997-cc0f-45b0-9816-887a62dde1e4" containerName="revoke-keys" Nov 01 06:44:18.469909 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:18.469900 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="97719997-cc0f-45b0-9816-887a62dde1e4" containerName="revoke-keys" Nov 01 06:44:18.473306 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:18.473289 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v" Nov 01 06:44:18.496148 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:18.496115 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg724\" (UniqueName: \"kubernetes.io/projected/bfa4234e-3def-4d10-b181-c675e28e1b36-kube-api-access-lg724\") pod \"sub-test-model-d7d444-kserve-7578986f47-75r8v\" (UID: \"bfa4234e-3def-4d10-b181-c675e28e1b36\") " pod="ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v" Nov 01 06:44:18.496367 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:18.496174 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfa4234e-3def-4d10-b181-c675e28e1b36-home\") pod \"sub-test-model-d7d444-kserve-7578986f47-75r8v\" (UID: \"bfa4234e-3def-4d10-b181-c675e28e1b36\") " pod="ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v" Nov 01 06:44:18.496367 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:18.496244 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfa4234e-3def-4d10-b181-c675e28e1b36-model-cache\") pod \"sub-test-model-d7d444-kserve-7578986f47-75r8v\" (UID: \"bfa4234e-3def-4d10-b181-c675e28e1b36\") " pod="ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v" Nov 01 06:44:18.496367 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:18.496345 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfa4234e-3def-4d10-b181-c675e28e1b36-tmp-dir\") pod \"sub-test-model-d7d444-kserve-7578986f47-75r8v\" (UID: \"bfa4234e-3def-4d10-b181-c675e28e1b36\") " pod="ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v" Nov 01 06:44:18.496568 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:18.496400 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfa4234e-3def-4d10-b181-c675e28e1b36-tls-certs\") pod \"sub-test-model-d7d444-kserve-7578986f47-75r8v\" (UID: \"bfa4234e-3def-4d10-b181-c675e28e1b36\") " pod="ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v" Nov 01 06:44:18.496568 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:18.496454 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfa4234e-3def-4d10-b181-c675e28e1b36-dshm\") pod \"sub-test-model-d7d444-kserve-7578986f47-75r8v\" (UID: \"bfa4234e-3def-4d10-b181-c675e28e1b36\") " pod="ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v" Nov 01 06:44:18.578716 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:18.578675 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-d7d444\"/\"default-dockercfg-rz78w\"" Nov 01 06:44:18.578716 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:18.578678 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-d7d444\"/\"openshift-service-ca.crt\"" Nov 01 06:44:18.578716 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:18.578721 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-d7d444\"/\"kube-root-ca.crt\"" Nov 01 06:44:18.579050 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:18.578747 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-d7d444\"/\"sub-test-model-d7d444-kserve-self-signed-certs\"" Nov 01 06:44:18.581994 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:18.581966 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v"] Nov 01 06:44:18.597714 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:18.597677 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg724\" (UniqueName: \"kubernetes.io/projected/bfa4234e-3def-4d10-b181-c675e28e1b36-kube-api-access-lg724\") pod \"sub-test-model-d7d444-kserve-7578986f47-75r8v\" (UID: \"bfa4234e-3def-4d10-b181-c675e28e1b36\") " pod="ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v" Nov 01 06:44:18.597890 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:18.597755 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfa4234e-3def-4d10-b181-c675e28e1b36-home\") pod \"sub-test-model-d7d444-kserve-7578986f47-75r8v\" (UID: \"bfa4234e-3def-4d10-b181-c675e28e1b36\") " pod="ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v" Nov 01 06:44:18.597890 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:18.597782 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfa4234e-3def-4d10-b181-c675e28e1b36-model-cache\") pod \"sub-test-model-d7d444-kserve-7578986f47-75r8v\" (UID: \"bfa4234e-3def-4d10-b181-c675e28e1b36\") " pod="ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v" Nov 01 06:44:18.597890 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:18.597809 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfa4234e-3def-4d10-b181-c675e28e1b36-tmp-dir\") pod \"sub-test-model-d7d444-kserve-7578986f47-75r8v\" (UID: \"bfa4234e-3def-4d10-b181-c675e28e1b36\") " pod="ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v" Nov 01 06:44:18.597890 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:18.597852 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfa4234e-3def-4d10-b181-c675e28e1b36-tls-certs\") pod \"sub-test-model-d7d444-kserve-7578986f47-75r8v\" (UID: \"bfa4234e-3def-4d10-b181-c675e28e1b36\") " pod="ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v" Nov 01 06:44:18.598152 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:18.597957 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfa4234e-3def-4d10-b181-c675e28e1b36-dshm\") pod \"sub-test-model-d7d444-kserve-7578986f47-75r8v\" (UID: \"bfa4234e-3def-4d10-b181-c675e28e1b36\") " pod="ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v" Nov 01 06:44:18.598215 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:18.598186 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfa4234e-3def-4d10-b181-c675e28e1b36-model-cache\") pod \"sub-test-model-d7d444-kserve-7578986f47-75r8v\" (UID: \"bfa4234e-3def-4d10-b181-c675e28e1b36\") " pod="ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v" Nov 01 06:44:18.598350 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:18.598326 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfa4234e-3def-4d10-b181-c675e28e1b36-tmp-dir\") pod \"sub-test-model-d7d444-kserve-7578986f47-75r8v\" (UID: \"bfa4234e-3def-4d10-b181-c675e28e1b36\") " pod="ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v" Nov 01 06:44:18.598512 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:18.598473 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfa4234e-3def-4d10-b181-c675e28e1b36-home\") pod \"sub-test-model-d7d444-kserve-7578986f47-75r8v\" (UID: \"bfa4234e-3def-4d10-b181-c675e28e1b36\") " pod="ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v" Nov 01 06:44:18.600329 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:18.600294 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfa4234e-3def-4d10-b181-c675e28e1b36-dshm\") pod \"sub-test-model-d7d444-kserve-7578986f47-75r8v\" (UID: \"bfa4234e-3def-4d10-b181-c675e28e1b36\") " pod="ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v" Nov 01 06:44:18.600712 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:18.600690 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfa4234e-3def-4d10-b181-c675e28e1b36-tls-certs\") pod \"sub-test-model-d7d444-kserve-7578986f47-75r8v\" (UID: \"bfa4234e-3def-4d10-b181-c675e28e1b36\") " pod="ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v" Nov 01 06:44:18.659306 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:18.659268 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg724\" (UniqueName: \"kubernetes.io/projected/bfa4234e-3def-4d10-b181-c675e28e1b36-kube-api-access-lg724\") pod \"sub-test-model-d7d444-kserve-7578986f47-75r8v\" (UID: \"bfa4234e-3def-4d10-b181-c675e28e1b36\") " pod="ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v" Nov 01 06:44:18.785956 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:18.785921 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v" Nov 01 06:44:19.176229 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:19.176091 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v"] Nov 01 06:44:19.179668 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:44:19.179637 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbfa4234e_3def_4d10_b181_c675e28e1b36.slice/crio-f17fdd3c6bba2f6a069ca8ff6e78cd47104f1e1e2724652294532261576c2047 WatchSource:0}: Error finding container f17fdd3c6bba2f6a069ca8ff6e78cd47104f1e1e2724652294532261576c2047: Status 404 returned error can't find the container with id f17fdd3c6bba2f6a069ca8ff6e78cd47104f1e1e2724652294532261576c2047 Nov 01 06:44:19.728585 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:19.728538 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v" event={"ID":"bfa4234e-3def-4d10-b181-c675e28e1b36","Type":"ContainerStarted","Data":"6e2991c7efe989dc8c919885bb3d870579049ad0272850eb73a49c3d6a2f06fb"} Nov 01 06:44:19.729008 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:19.728599 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v" event={"ID":"bfa4234e-3def-4d10-b181-c675e28e1b36","Type":"ContainerStarted","Data":"f17fdd3c6bba2f6a069ca8ff6e78cd47104f1e1e2724652294532261576c2047"} Nov 01 06:44:19.786664 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:19.786629 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v" Nov 01 06:44:19.847966 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:19.847908 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v" podStartSLOduration=1.847894742 podStartE2EDuration="1.847894742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:44:19.845441666 +0000 UTC m=+3538.500558838" watchObservedRunningTime="2026-11-01 06:44:19.847894742 +0000 UTC m=+3538.503011937" Nov 01 06:44:27.601626 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:27.601590 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f00072c-vrlxg"] Nov 01 06:44:27.605293 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:27.605275 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f00072c-vrlxg" Nov 01 06:44:27.708922 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:27.708891 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:44:27.709133 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:27.708957 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:44:27.709133 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:27.708978 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:44:27.709133 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:27.709012 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:44:27.712822 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:27.712794 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f00072c-vrlxg"] Nov 01 06:44:27.724145 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:27.724119 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b720868-e423-4738-bb3b-80f505d1bf2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f00072c-vrlxg\" (UID: \"0b720868-e423-4738-bb3b-80f505d1bf2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f00072c-vrlxg" Nov 01 06:44:27.724286 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:27.724190 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b720868-e423-4738-bb3b-80f505d1bf2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f00072c-vrlxg\" (UID: \"0b720868-e423-4738-bb3b-80f505d1bf2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f00072c-vrlxg" Nov 01 06:44:27.724342 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:27.724282 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdk6v\" (UniqueName: \"kubernetes.io/projected/0b720868-e423-4738-bb3b-80f505d1bf2b-kube-api-access-vdk6v\") pod \"maas-api-revoke-sub-2f00072c-vrlxg\" (UID: \"0b720868-e423-4738-bb3b-80f505d1bf2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f00072c-vrlxg" Nov 01 06:44:27.825926 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:27.825873 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b720868-e423-4738-bb3b-80f505d1bf2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f00072c-vrlxg\" (UID: \"0b720868-e423-4738-bb3b-80f505d1bf2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f00072c-vrlxg" Nov 01 06:44:27.826173 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:27.825998 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b720868-e423-4738-bb3b-80f505d1bf2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f00072c-vrlxg\" (UID: \"0b720868-e423-4738-bb3b-80f505d1bf2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f00072c-vrlxg" Nov 01 06:44:27.826173 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:27.826034 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdk6v\" (UniqueName: \"kubernetes.io/projected/0b720868-e423-4738-bb3b-80f505d1bf2b-kube-api-access-vdk6v\") pod \"maas-api-revoke-sub-2f00072c-vrlxg\" (UID: \"0b720868-e423-4738-bb3b-80f505d1bf2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f00072c-vrlxg" Nov 01 06:44:27.826624 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:27.826598 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b720868-e423-4738-bb3b-80f505d1bf2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f00072c-vrlxg\" (UID: \"0b720868-e423-4738-bb3b-80f505d1bf2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f00072c-vrlxg" Nov 01 06:44:27.828398 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:27.828377 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b720868-e423-4738-bb3b-80f505d1bf2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f00072c-vrlxg\" (UID: \"0b720868-e423-4738-bb3b-80f505d1bf2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f00072c-vrlxg" Nov 01 06:44:27.887865 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:27.887776 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdk6v\" (UniqueName: \"kubernetes.io/projected/0b720868-e423-4738-bb3b-80f505d1bf2b-kube-api-access-vdk6v\") pod \"maas-api-revoke-sub-2f00072c-vrlxg\" (UID: \"0b720868-e423-4738-bb3b-80f505d1bf2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f00072c-vrlxg" Nov 01 06:44:27.918651 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:27.918622 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f00072c-vrlxg" Nov 01 06:44:28.296531 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:28.296489 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f00072c-vrlxg"] Nov 01 06:44:28.297240 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:44:28.297218 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b720868_e423_4738_bb3b_80f505d1bf2b.slice/crio-834505776c69722e2d298b3b33047fa14458ba68c40633857d68fa79a1247260 WatchSource:0}: Error finding container 834505776c69722e2d298b3b33047fa14458ba68c40633857d68fa79a1247260: Status 404 returned error can't find the container with id 834505776c69722e2d298b3b33047fa14458ba68c40633857d68fa79a1247260 Nov 01 06:44:28.786328 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:28.786295 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v" Nov 01 06:44:28.789554 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:28.789519 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f00072c-vrlxg" event={"ID":"0b720868-e423-4738-bb3b-80f505d1bf2b","Type":"ContainerStarted","Data":"1acd03eaae2dcd1ebb9264f6e36a483f2c07c834f3e5fdd208c229ee7f72e804"} Nov 01 06:44:28.789757 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:28.789582 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f00072c-vrlxg" event={"ID":"0b720868-e423-4738-bb3b-80f505d1bf2b","Type":"ContainerStarted","Data":"834505776c69722e2d298b3b33047fa14458ba68c40633857d68fa79a1247260"} Nov 01 06:44:28.799772 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:28.799749 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v" Nov 01 06:44:28.908790 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:28.908680 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f00072c-vrlxg" podStartSLOduration=1.908662684 podStartE2EDuration="1.908662684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:44:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:44:28.9064294 +0000 UTC m=+3547.561546568" watchObservedRunningTime="2026-11-01 06:44:28.908662684 +0000 UTC m=+3547.563779850" Nov 01 06:44:29.797898 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:29.797857 5040 generic.go:356] "Generic (PLEG): container finished" podID="0b720868-e423-4738-bb3b-80f505d1bf2b" containerID="1acd03eaae2dcd1ebb9264f6e36a483f2c07c834f3e5fdd208c229ee7f72e804" exitCode=0 Nov 01 06:44:29.798388 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:29.797944 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f00072c-vrlxg" event={"ID":"0b720868-e423-4738-bb3b-80f505d1bf2b","Type":"ContainerDied","Data":"1acd03eaae2dcd1ebb9264f6e36a483f2c07c834f3e5fdd208c229ee7f72e804"} Nov 01 06:44:29.798472 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:29.798450 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v" Nov 01 06:44:30.953631 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:30.953605 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f00072c-vrlxg" Nov 01 06:44:31.062250 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:31.062163 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b720868-e423-4738-bb3b-80f505d1bf2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b720868-e423-4738-bb3b-80f505d1bf2b-maas-api-xks-ca\") pod \"0b720868-e423-4738-bb3b-80f505d1bf2b\" (UID: \"0b720868-e423-4738-bb3b-80f505d1bf2b\") " Nov 01 06:44:31.062250 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:31.062234 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b720868-e423-4738-bb3b-80f505d1bf2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b720868-e423-4738-bb3b-80f505d1bf2b-maas-api-openshift-ca\") pod \"0b720868-e423-4738-bb3b-80f505d1bf2b\" (UID: \"0b720868-e423-4738-bb3b-80f505d1bf2b\") " Nov 01 06:44:31.062532 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:31.062340 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b720868-e423-4738-bb3b-80f505d1bf2b-kube-api-access-vdk6v\" (UniqueName: \"kubernetes.io/projected/0b720868-e423-4738-bb3b-80f505d1bf2b-kube-api-access-vdk6v\") pod \"0b720868-e423-4738-bb3b-80f505d1bf2b\" (UID: \"0b720868-e423-4738-bb3b-80f505d1bf2b\") " Nov 01 06:44:31.062662 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:31.062637 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b720868-e423-4738-bb3b-80f505d1bf2b-maas-api-openshift-ca" pod "0b720868-e423-4738-bb3b-80f505d1bf2b" (UID: "0b720868-e423-4738-bb3b-80f505d1bf2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:44:31.062793 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:31.062767 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b720868-e423-4738-bb3b-80f505d1bf2b-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:44:31.064429 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:31.064407 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b720868-e423-4738-bb3b-80f505d1bf2b-maas-api-xks-ca" pod "0b720868-e423-4738-bb3b-80f505d1bf2b" (UID: "0b720868-e423-4738-bb3b-80f505d1bf2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:44:31.064548 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:31.064524 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b720868-e423-4738-bb3b-80f505d1bf2b-kube-api-access-vdk6v" pod "0b720868-e423-4738-bb3b-80f505d1bf2b" (UID: "0b720868-e423-4738-bb3b-80f505d1bf2b"). InnerVolumeSpecName "kube-api-access-vdk6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:44:31.164004 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:31.163950 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdk6v\" (UniqueName: \"kubernetes.io/projected/0b720868-e423-4738-bb3b-80f505d1bf2b-kube-api-access-vdk6v\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:44:31.164004 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:31.164001 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b720868-e423-4738-bb3b-80f505d1bf2b-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:44:31.813612 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:31.813531 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f00072c-vrlxg" Nov 01 06:44:31.813770 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:31.813531 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f00072c-vrlxg" event={"ID":"0b720868-e423-4738-bb3b-80f505d1bf2b","Type":"ContainerDied","Data":"834505776c69722e2d298b3b33047fa14458ba68c40633857d68fa79a1247260"} Nov 01 06:44:31.813770 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:31.813665 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="834505776c69722e2d298b3b33047fa14458ba68c40633857d68fa79a1247260" Nov 01 06:44:39.973803 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:44:39.973752 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:44:51.084279 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:51.084237 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56"] Nov 01 06:44:51.084996 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:51.084972 5040 cpu_manager.go:414] "containerMap: removing container" podUID="0b720868-e423-4738-bb3b-80f505d1bf2b" containerName="revoke-keys" Nov 01 06:44:51.085092 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:51.084998 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b720868-e423-4738-bb3b-80f505d1bf2b" containerName="revoke-keys" Nov 01 06:44:51.085183 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:51.085170 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b720868-e423-4738-bb3b-80f505d1bf2b" containerName="revoke-keys" Nov 01 06:44:51.088650 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:51.088625 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56" Nov 01 06:44:51.159209 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:51.159176 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56"] Nov 01 06:44:51.192574 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:51.192528 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-747b7b\"/\"openshift-service-ca.crt\"" Nov 01 06:44:51.192574 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:51.192528 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-747b7b\"/\"default-dockercfg-m7px6\"" Nov 01 06:44:51.192574 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:51.192544 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-747b7b\"/\"kube-root-ca.crt\"" Nov 01 06:44:51.192828 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:51.192635 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-747b7b\"/\"sub-test-model-747b7b-kserve-self-signed-certs\"" Nov 01 06:44:51.215406 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:51.215367 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvpb9\" (UniqueName: \"kubernetes.io/projected/1fbb1642-46b7-4489-bdac-2075d11d644e-kube-api-access-wvpb9\") pod \"sub-test-model-747b7b-kserve-65f6846b4c-wsx56\" (UID: \"1fbb1642-46b7-4489-bdac-2075d11d644e\") " pod="ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56" Nov 01 06:44:51.215609 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:51.215414 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fbb1642-46b7-4489-bdac-2075d11d644e-home\") pod \"sub-test-model-747b7b-kserve-65f6846b4c-wsx56\" (UID: \"1fbb1642-46b7-4489-bdac-2075d11d644e\") " pod="ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56" Nov 01 06:44:51.215609 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:51.215508 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fbb1642-46b7-4489-bdac-2075d11d644e-tmp-dir\") pod \"sub-test-model-747b7b-kserve-65f6846b4c-wsx56\" (UID: \"1fbb1642-46b7-4489-bdac-2075d11d644e\") " pod="ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56" Nov 01 06:44:51.215609 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:51.215552 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fbb1642-46b7-4489-bdac-2075d11d644e-tls-certs\") pod \"sub-test-model-747b7b-kserve-65f6846b4c-wsx56\" (UID: \"1fbb1642-46b7-4489-bdac-2075d11d644e\") " pod="ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56" Nov 01 06:44:51.215744 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:51.215664 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fbb1642-46b7-4489-bdac-2075d11d644e-model-cache\") pod \"sub-test-model-747b7b-kserve-65f6846b4c-wsx56\" (UID: \"1fbb1642-46b7-4489-bdac-2075d11d644e\") " pod="ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56" Nov 01 06:44:51.215744 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:51.215730 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fbb1642-46b7-4489-bdac-2075d11d644e-dshm\") pod \"sub-test-model-747b7b-kserve-65f6846b4c-wsx56\" (UID: \"1fbb1642-46b7-4489-bdac-2075d11d644e\") " pod="ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56" Nov 01 06:44:51.317187 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:51.317132 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fbb1642-46b7-4489-bdac-2075d11d644e-model-cache\") pod \"sub-test-model-747b7b-kserve-65f6846b4c-wsx56\" (UID: \"1fbb1642-46b7-4489-bdac-2075d11d644e\") " pod="ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56" Nov 01 06:44:51.317409 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:51.317263 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fbb1642-46b7-4489-bdac-2075d11d644e-dshm\") pod \"sub-test-model-747b7b-kserve-65f6846b4c-wsx56\" (UID: \"1fbb1642-46b7-4489-bdac-2075d11d644e\") " pod="ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56" Nov 01 06:44:51.317490 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:51.317452 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvpb9\" (UniqueName: \"kubernetes.io/projected/1fbb1642-46b7-4489-bdac-2075d11d644e-kube-api-access-wvpb9\") pod \"sub-test-model-747b7b-kserve-65f6846b4c-wsx56\" (UID: \"1fbb1642-46b7-4489-bdac-2075d11d644e\") " pod="ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56" Nov 01 06:44:51.317555 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:51.317539 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fbb1642-46b7-4489-bdac-2075d11d644e-home\") pod \"sub-test-model-747b7b-kserve-65f6846b4c-wsx56\" (UID: \"1fbb1642-46b7-4489-bdac-2075d11d644e\") " pod="ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56" Nov 01 06:44:51.317611 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:51.317594 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fbb1642-46b7-4489-bdac-2075d11d644e-tmp-dir\") pod \"sub-test-model-747b7b-kserve-65f6846b4c-wsx56\" (UID: \"1fbb1642-46b7-4489-bdac-2075d11d644e\") " pod="ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56" Nov 01 06:44:51.317667 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:51.317628 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fbb1642-46b7-4489-bdac-2075d11d644e-model-cache\") pod \"sub-test-model-747b7b-kserve-65f6846b4c-wsx56\" (UID: \"1fbb1642-46b7-4489-bdac-2075d11d644e\") " pod="ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56" Nov 01 06:44:51.317667 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:51.317636 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fbb1642-46b7-4489-bdac-2075d11d644e-tls-certs\") pod \"sub-test-model-747b7b-kserve-65f6846b4c-wsx56\" (UID: \"1fbb1642-46b7-4489-bdac-2075d11d644e\") " pod="ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56" Nov 01 06:44:51.317927 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:51.317904 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fbb1642-46b7-4489-bdac-2075d11d644e-home\") pod \"sub-test-model-747b7b-kserve-65f6846b4c-wsx56\" (UID: \"1fbb1642-46b7-4489-bdac-2075d11d644e\") " pod="ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56" Nov 01 06:44:51.318590 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:51.318554 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fbb1642-46b7-4489-bdac-2075d11d644e-tmp-dir\") pod \"sub-test-model-747b7b-kserve-65f6846b4c-wsx56\" (UID: \"1fbb1642-46b7-4489-bdac-2075d11d644e\") " pod="ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56" Nov 01 06:44:51.320074 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:51.320044 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fbb1642-46b7-4489-bdac-2075d11d644e-dshm\") pod \"sub-test-model-747b7b-kserve-65f6846b4c-wsx56\" (UID: \"1fbb1642-46b7-4489-bdac-2075d11d644e\") " pod="ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56" Nov 01 06:44:51.321078 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:51.321051 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fbb1642-46b7-4489-bdac-2075d11d644e-tls-certs\") pod \"sub-test-model-747b7b-kserve-65f6846b4c-wsx56\" (UID: \"1fbb1642-46b7-4489-bdac-2075d11d644e\") " pod="ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56" Nov 01 06:44:51.380099 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:51.380019 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvpb9\" (UniqueName: \"kubernetes.io/projected/1fbb1642-46b7-4489-bdac-2075d11d644e-kube-api-access-wvpb9\") pod \"sub-test-model-747b7b-kserve-65f6846b4c-wsx56\" (UID: \"1fbb1642-46b7-4489-bdac-2075d11d644e\") " pod="ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56" Nov 01 06:44:51.401103 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:51.401065 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56" Nov 01 06:44:51.802883 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:44:51.802851 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fbb1642_46b7_4489_bdac_2075d11d644e.slice/crio-2759774f16906f1a736ca879d65d960bb332bd45282ae095b3199fff719d24d4 WatchSource:0}: Error finding container 2759774f16906f1a736ca879d65d960bb332bd45282ae095b3199fff719d24d4: Status 404 returned error can't find the container with id 2759774f16906f1a736ca879d65d960bb332bd45282ae095b3199fff719d24d4 Nov 01 06:44:51.803029 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:51.802944 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56"] Nov 01 06:44:51.972074 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:51.972028 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56" event={"ID":"1fbb1642-46b7-4489-bdac-2075d11d644e","Type":"ContainerStarted","Data":"1cb2a402c100bf19e79a92b3f0001cca0baabc131df0f1a40e5d33ba717ca9e9"} Nov 01 06:44:51.972074 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:51.972077 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56" event={"ID":"1fbb1642-46b7-4489-bdac-2075d11d644e","Type":"ContainerStarted","Data":"2759774f16906f1a736ca879d65d960bb332bd45282ae095b3199fff719d24d4"} Nov 01 06:44:52.091714 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:52.091604 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56" podStartSLOduration=1.091588537 podStartE2EDuration="1.091588537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:44:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:44:52.089268204 +0000 UTC m=+3570.744385371" watchObservedRunningTime="2026-11-01 06:44:52.091588537 +0000 UTC m=+3570.746705762" Nov 01 06:44:52.401671 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:44:52.401584 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56" Nov 01 06:45:00.163377 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.163337 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-747b7b-29891925-g8kd9"] Nov 01 06:45:00.172379 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.172334 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-747b7b-29891925-g8kd9" Nov 01 06:45:00.174884 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.174857 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891925-5r4hb"] Nov 01 06:45:00.180099 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.180075 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891925-zh28x"] Nov 01 06:45:00.180219 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.180190 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891925-5r4hb" Nov 01 06:45:00.184736 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.184710 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891925-zh28x" Nov 01 06:45:00.213470 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.213441 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891925-chf96"] Nov 01 06:45:00.217149 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.217128 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891925-dqkfq"] Nov 01 06:45:00.217264 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.217231 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891925-chf96" Nov 01 06:45:00.221003 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.220984 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891925-cx2hq"] Nov 01 06:45:00.221125 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.221105 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891925-dqkfq" Nov 01 06:45:00.224596 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.224579 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5be4fa-29891925-l24ff"] Nov 01 06:45:00.224716 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.224701 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891925-cx2hq" Nov 01 06:45:00.226023 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.226002 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:45:00.228235 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.228220 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891925-c7wn9"] Nov 01 06:45:00.228378 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.228361 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5be4fa-29891925-l24ff" Nov 01 06:45:00.229839 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.229816 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvnd8\" (UniqueName: \"kubernetes.io/projected/e328a3ce-7f5d-4599-a616-b49f6f18a510-kube-api-access-fvnd8\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-747b7b-29891925-g8kd9\" (UID: \"e328a3ce-7f5d-4599-a616-b49f6f18a510\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-747b7b-29891925-g8kd9" Nov 01 06:45:00.230018 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.229998 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e328a3ce-7f5d-4599-a616-b49f6f18a510-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-747b7b-29891925-g8kd9\" (UID: \"e328a3ce-7f5d-4599-a616-b49f6f18a510\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-747b7b-29891925-g8kd9" Nov 01 06:45:00.231896 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.231878 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891925-c7wn9" Nov 01 06:45:00.260273 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.260255 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:45:00.268047 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.268024 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891925-5r4hb"] Nov 01 06:45:00.271222 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.271203 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-747b7b-29891925-g8kd9"] Nov 01 06:45:00.317516 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.317418 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891925-chf96"] Nov 01 06:45:00.324077 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.322680 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:45:00.324077 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.323695 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891925-dqkfq"] Nov 01 06:45:00.330144 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.330119 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891925-cx2hq"] Nov 01 06:45:00.330690 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.330672 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7482f875-ca8d-4b85-ab43-5e7eea31efca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-3986bf-29891925-dqkfq\" (UID: \"7482f875-ca8d-4b85-ab43-5e7eea31efca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891925-dqkfq" Nov 01 06:45:00.330745 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.330705 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c4bca9d-4e46-44e4-9270-ea42ada707bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891925-cx2hq\" (UID: \"1c4bca9d-4e46-44e4-9270-ea42ada707bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891925-cx2hq" Nov 01 06:45:00.330745 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.330736 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a8d4067-a520-4890-9d89-e73e382438b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891925-zh28x\" (UID: \"1a8d4067-a520-4890-9d89-e73e382438b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891925-zh28x" Nov 01 06:45:00.330823 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.330755 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cjh9\" (UniqueName: \"kubernetes.io/projected/7482f875-ca8d-4b85-ab43-5e7eea31efca-kube-api-access-4cjh9\") pod \"maas-api-key-cleanup-e2e-extmod-b-3986bf-29891925-dqkfq\" (UID: \"7482f875-ca8d-4b85-ab43-5e7eea31efca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891925-dqkfq" Nov 01 06:45:00.330823 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.330779 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5566m\" (UniqueName: \"kubernetes.io/projected/c57c77c0-2554-4f46-9c1a-a8321a993953-kube-api-access-5566m\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891925-chf96\" (UID: \"c57c77c0-2554-4f46-9c1a-a8321a993953\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891925-chf96" Nov 01 06:45:00.330823 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.330804 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e328a3ce-7f5d-4599-a616-b49f6f18a510-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-747b7b-29891925-g8kd9\" (UID: \"e328a3ce-7f5d-4599-a616-b49f6f18a510\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-747b7b-29891925-g8kd9" Nov 01 06:45:00.330823 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.330821 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4957cc07-721f-40ca-85e7-3d9ca330096e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-5be4fa-29891925-l24ff\" (UID: \"4957cc07-721f-40ca-85e7-3d9ca330096e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5be4fa-29891925-l24ff" Nov 01 06:45:00.330950 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.330845 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpsgx\" (UniqueName: \"kubernetes.io/projected/0431f2e8-f3aa-4f41-9360-0f12c5baa029-kube-api-access-tpsgx\") pod \"maas-api-key-cleanup-e2e-extmod-a-cfe368-29891925-5r4hb\" (UID: \"0431f2e8-f3aa-4f41-9360-0f12c5baa029\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891925-5r4hb" Nov 01 06:45:00.330950 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.330863 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891925-c7wn9\" (UID: \"d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891925-c7wn9" Nov 01 06:45:00.330950 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.330888 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0431f2e8-f3aa-4f41-9360-0f12c5baa029-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-cfe368-29891925-5r4hb\" (UID: \"0431f2e8-f3aa-4f41-9360-0f12c5baa029\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891925-5r4hb" Nov 01 06:45:00.330950 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.330929 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjf68\" (UniqueName: \"kubernetes.io/projected/4957cc07-721f-40ca-85e7-3d9ca330096e-kube-api-access-fjf68\") pod \"maas-api-key-cleanup-e2e-extmod-a-5be4fa-29891925-l24ff\" (UID: \"4957cc07-721f-40ca-85e7-3d9ca330096e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5be4fa-29891925-l24ff" Nov 01 06:45:00.331080 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.330956 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmtmp\" (UniqueName: \"kubernetes.io/projected/1a8d4067-a520-4890-9d89-e73e382438b1-kube-api-access-rmtmp\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891925-zh28x\" (UID: \"1a8d4067-a520-4890-9d89-e73e382438b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891925-zh28x" Nov 01 06:45:00.331080 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.330976 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqgzp\" (UniqueName: \"kubernetes.io/projected/1c4bca9d-4e46-44e4-9270-ea42ada707bb-kube-api-access-jqgzp\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891925-cx2hq\" (UID: \"1c4bca9d-4e46-44e4-9270-ea42ada707bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891925-cx2hq" Nov 01 06:45:00.331080 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.331041 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvnd8\" (UniqueName: \"kubernetes.io/projected/e328a3ce-7f5d-4599-a616-b49f6f18a510-kube-api-access-fvnd8\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-747b7b-29891925-g8kd9\" (UID: \"e328a3ce-7f5d-4599-a616-b49f6f18a510\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-747b7b-29891925-g8kd9" Nov 01 06:45:00.331080 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.331056 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57c77c0-2554-4f46-9c1a-a8321a993953-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891925-chf96\" (UID: \"c57c77c0-2554-4f46-9c1a-a8321a993953\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891925-chf96" Nov 01 06:45:00.331207 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.331082 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb722\" (UniqueName: \"kubernetes.io/projected/d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9-kube-api-access-gb722\") pod \"maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891925-c7wn9\" (UID: \"d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891925-c7wn9" Nov 01 06:45:00.331817 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.331795 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e328a3ce-7f5d-4599-a616-b49f6f18a510-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-747b7b-29891925-g8kd9\" (UID: \"e328a3ce-7f5d-4599-a616-b49f6f18a510\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-747b7b-29891925-g8kd9" Nov 01 06:45:00.334519 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.334495 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5be4fa-29891925-l24ff"] Nov 01 06:45:00.371971 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.371933 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891925-zh28x"] Nov 01 06:45:00.388527 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.388406 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891925-c7wn9"] Nov 01 06:45:00.394705 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.394677 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvnd8\" (UniqueName: \"kubernetes.io/projected/e328a3ce-7f5d-4599-a616-b49f6f18a510-kube-api-access-fvnd8\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-747b7b-29891925-g8kd9\" (UID: \"e328a3ce-7f5d-4599-a616-b49f6f18a510\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-747b7b-29891925-g8kd9" Nov 01 06:45:00.431765 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.431678 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5566m\" (UniqueName: \"kubernetes.io/projected/c57c77c0-2554-4f46-9c1a-a8321a993953-kube-api-access-5566m\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891925-chf96\" (UID: \"c57c77c0-2554-4f46-9c1a-a8321a993953\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891925-chf96" Nov 01 06:45:00.431765 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.431722 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4957cc07-721f-40ca-85e7-3d9ca330096e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-5be4fa-29891925-l24ff\" (UID: \"4957cc07-721f-40ca-85e7-3d9ca330096e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5be4fa-29891925-l24ff" Nov 01 06:45:00.431964 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.431847 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpsgx\" (UniqueName: \"kubernetes.io/projected/0431f2e8-f3aa-4f41-9360-0f12c5baa029-kube-api-access-tpsgx\") pod \"maas-api-key-cleanup-e2e-extmod-a-cfe368-29891925-5r4hb\" (UID: \"0431f2e8-f3aa-4f41-9360-0f12c5baa029\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891925-5r4hb" Nov 01 06:45:00.431964 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.431883 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891925-c7wn9\" (UID: \"d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891925-c7wn9" Nov 01 06:45:00.431964 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.431930 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0431f2e8-f3aa-4f41-9360-0f12c5baa029-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-cfe368-29891925-5r4hb\" (UID: \"0431f2e8-f3aa-4f41-9360-0f12c5baa029\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891925-5r4hb" Nov 01 06:45:00.432110 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.431990 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjf68\" (UniqueName: \"kubernetes.io/projected/4957cc07-721f-40ca-85e7-3d9ca330096e-kube-api-access-fjf68\") pod \"maas-api-key-cleanup-e2e-extmod-a-5be4fa-29891925-l24ff\" (UID: \"4957cc07-721f-40ca-85e7-3d9ca330096e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5be4fa-29891925-l24ff" Nov 01 06:45:00.432110 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.432036 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmtmp\" (UniqueName: \"kubernetes.io/projected/1a8d4067-a520-4890-9d89-e73e382438b1-kube-api-access-rmtmp\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891925-zh28x\" (UID: \"1a8d4067-a520-4890-9d89-e73e382438b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891925-zh28x" Nov 01 06:45:00.432110 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.432070 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqgzp\" (UniqueName: \"kubernetes.io/projected/1c4bca9d-4e46-44e4-9270-ea42ada707bb-kube-api-access-jqgzp\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891925-cx2hq\" (UID: \"1c4bca9d-4e46-44e4-9270-ea42ada707bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891925-cx2hq" Nov 01 06:45:00.432282 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.432178 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57c77c0-2554-4f46-9c1a-a8321a993953-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891925-chf96\" (UID: \"c57c77c0-2554-4f46-9c1a-a8321a993953\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891925-chf96" Nov 01 06:45:00.432353 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.432305 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb722\" (UniqueName: \"kubernetes.io/projected/d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9-kube-api-access-gb722\") pod \"maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891925-c7wn9\" (UID: \"d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891925-c7wn9" Nov 01 06:45:00.432413 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.432378 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7482f875-ca8d-4b85-ab43-5e7eea31efca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-3986bf-29891925-dqkfq\" (UID: \"7482f875-ca8d-4b85-ab43-5e7eea31efca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891925-dqkfq" Nov 01 06:45:00.432470 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.432420 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c4bca9d-4e46-44e4-9270-ea42ada707bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891925-cx2hq\" (UID: \"1c4bca9d-4e46-44e4-9270-ea42ada707bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891925-cx2hq" Nov 01 06:45:00.432580 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.432492 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a8d4067-a520-4890-9d89-e73e382438b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891925-zh28x\" (UID: \"1a8d4067-a520-4890-9d89-e73e382438b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891925-zh28x" Nov 01 06:45:00.432580 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.432524 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cjh9\" (UniqueName: \"kubernetes.io/projected/7482f875-ca8d-4b85-ab43-5e7eea31efca-kube-api-access-4cjh9\") pod \"maas-api-key-cleanup-e2e-extmod-b-3986bf-29891925-dqkfq\" (UID: \"7482f875-ca8d-4b85-ab43-5e7eea31efca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891925-dqkfq" Nov 01 06:45:00.432580 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.432553 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4957cc07-721f-40ca-85e7-3d9ca330096e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-5be4fa-29891925-l24ff\" (UID: \"4957cc07-721f-40ca-85e7-3d9ca330096e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5be4fa-29891925-l24ff" Nov 01 06:45:00.432870 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.432837 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891925-c7wn9\" (UID: \"d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891925-c7wn9" Nov 01 06:45:00.433282 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.433259 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7482f875-ca8d-4b85-ab43-5e7eea31efca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-3986bf-29891925-dqkfq\" (UID: \"7482f875-ca8d-4b85-ab43-5e7eea31efca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891925-dqkfq" Nov 01 06:45:00.433401 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.433297 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c4bca9d-4e46-44e4-9270-ea42ada707bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891925-cx2hq\" (UID: \"1c4bca9d-4e46-44e4-9270-ea42ada707bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891925-cx2hq" Nov 01 06:45:00.433401 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.433344 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a8d4067-a520-4890-9d89-e73e382438b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891925-zh28x\" (UID: \"1a8d4067-a520-4890-9d89-e73e382438b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891925-zh28x" Nov 01 06:45:00.433577 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.433415 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57c77c0-2554-4f46-9c1a-a8321a993953-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891925-chf96\" (UID: \"c57c77c0-2554-4f46-9c1a-a8321a993953\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891925-chf96" Nov 01 06:45:00.433577 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.433542 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0431f2e8-f3aa-4f41-9360-0f12c5baa029-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-cfe368-29891925-5r4hb\" (UID: \"0431f2e8-f3aa-4f41-9360-0f12c5baa029\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891925-5r4hb" Nov 01 06:45:00.488171 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.488136 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-747b7b-29891925-g8kd9" Nov 01 06:45:00.494969 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.493890 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjf68\" (UniqueName: \"kubernetes.io/projected/4957cc07-721f-40ca-85e7-3d9ca330096e-kube-api-access-fjf68\") pod \"maas-api-key-cleanup-e2e-extmod-a-5be4fa-29891925-l24ff\" (UID: \"4957cc07-721f-40ca-85e7-3d9ca330096e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5be4fa-29891925-l24ff" Nov 01 06:45:00.494969 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.493933 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5566m\" (UniqueName: \"kubernetes.io/projected/c57c77c0-2554-4f46-9c1a-a8321a993953-kube-api-access-5566m\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891925-chf96\" (UID: \"c57c77c0-2554-4f46-9c1a-a8321a993953\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891925-chf96" Nov 01 06:45:00.494969 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.494138 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpsgx\" (UniqueName: \"kubernetes.io/projected/0431f2e8-f3aa-4f41-9360-0f12c5baa029-kube-api-access-tpsgx\") pod \"maas-api-key-cleanup-e2e-extmod-a-cfe368-29891925-5r4hb\" (UID: \"0431f2e8-f3aa-4f41-9360-0f12c5baa029\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891925-5r4hb" Nov 01 06:45:00.494969 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.494444 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb722\" (UniqueName: \"kubernetes.io/projected/d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9-kube-api-access-gb722\") pod \"maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891925-c7wn9\" (UID: \"d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891925-c7wn9" Nov 01 06:45:00.494969 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.494506 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cjh9\" (UniqueName: \"kubernetes.io/projected/7482f875-ca8d-4b85-ab43-5e7eea31efca-kube-api-access-4cjh9\") pod \"maas-api-key-cleanup-e2e-extmod-b-3986bf-29891925-dqkfq\" (UID: \"7482f875-ca8d-4b85-ab43-5e7eea31efca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891925-dqkfq" Nov 01 06:45:00.495224 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.495024 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmtmp\" (UniqueName: \"kubernetes.io/projected/1a8d4067-a520-4890-9d89-e73e382438b1-kube-api-access-rmtmp\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891925-zh28x\" (UID: \"1a8d4067-a520-4890-9d89-e73e382438b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891925-zh28x" Nov 01 06:45:00.495224 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.495083 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqgzp\" (UniqueName: \"kubernetes.io/projected/1c4bca9d-4e46-44e4-9270-ea42ada707bb-kube-api-access-jqgzp\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891925-cx2hq\" (UID: \"1c4bca9d-4e46-44e4-9270-ea42ada707bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891925-cx2hq" Nov 01 06:45:00.499539 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.498190 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891925-5r4hb" Nov 01 06:45:00.505471 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.505449 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891925-zh28x" Nov 01 06:45:00.532160 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.532112 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891925-chf96" Nov 01 06:45:00.539718 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.539685 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891925-dqkfq" Nov 01 06:45:00.547543 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.547331 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891925-cx2hq" Nov 01 06:45:00.557328 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.557130 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5be4fa-29891925-l24ff" Nov 01 06:45:00.564023 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.563366 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891925-c7wn9" Nov 01 06:45:00.914946 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:45:00.914911 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode328a3ce_7f5d_4599_a616_b49f6f18a510.slice/crio-5cd313306d2ba9d2b8cd81c97395e395012f4ad66222db5cd22f53b4e927b28a WatchSource:0}: Error finding container 5cd313306d2ba9d2b8cd81c97395e395012f4ad66222db5cd22f53b4e927b28a: Status 404 returned error can't find the container with id 5cd313306d2ba9d2b8cd81c97395e395012f4ad66222db5cd22f53b4e927b28a Nov 01 06:45:00.915411 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.915386 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-747b7b-29891925-g8kd9"] Nov 01 06:45:00.949098 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.949064 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891925-zh28x"] Nov 01 06:45:00.949755 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:45:00.949727 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8d4067_a520_4890_9d89_e73e382438b1.slice/crio-6989d20a8a88272b64b54c772839be735ab256e60c45f065d5135f50d9e6741d WatchSource:0}: Error finding container 6989d20a8a88272b64b54c772839be735ab256e60c45f065d5135f50d9e6741d: Status 404 returned error can't find the container with id 6989d20a8a88272b64b54c772839be735ab256e60c45f065d5135f50d9e6741d Nov 01 06:45:00.951272 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:45:00.951245 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc57c77c0_2554_4f46_9c1a_a8321a993953.slice/crio-a61dc48b12e7e4286b213acd9f38bcc3b6234e87cdd43db799d0366370bc40f4 WatchSource:0}: Error finding container a61dc48b12e7e4286b213acd9f38bcc3b6234e87cdd43db799d0366370bc40f4: Status 404 returned error can't find the container with id a61dc48b12e7e4286b213acd9f38bcc3b6234e87cdd43db799d0366370bc40f4 Nov 01 06:45:00.953789 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:00.953766 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891925-chf96"] Nov 01 06:45:01.011784 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:01.011744 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891925-dqkfq"] Nov 01 06:45:01.024888 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:01.024866 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891925-cx2hq"] Nov 01 06:45:01.033424 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:01.033388 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891925-chf96" event={"ID":"c57c77c0-2554-4f46-9c1a-a8321a993953","Type":"ContainerStarted","Data":"a61dc48b12e7e4286b213acd9f38bcc3b6234e87cdd43db799d0366370bc40f4"} Nov 01 06:45:01.034750 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:01.034708 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-747b7b-29891925-g8kd9" event={"ID":"e328a3ce-7f5d-4599-a616-b49f6f18a510","Type":"ContainerStarted","Data":"5cd313306d2ba9d2b8cd81c97395e395012f4ad66222db5cd22f53b4e927b28a"} Nov 01 06:45:01.036187 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:01.036164 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891925-zh28x" event={"ID":"1a8d4067-a520-4890-9d89-e73e382438b1","Type":"ContainerStarted","Data":"6989d20a8a88272b64b54c772839be735ab256e60c45f065d5135f50d9e6741d"} Nov 01 06:45:01.053474 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:01.053453 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891925-c7wn9"] Nov 01 06:45:01.104262 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:01.104237 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891925-5r4hb"] Nov 01 06:45:01.148247 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:01.148220 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5be4fa-29891925-l24ff"] Nov 01 06:45:01.195590 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:45:01.195503 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7482f875_ca8d_4b85_ab43_5e7eea31efca.slice/crio-2db9420c2bfd094fbd3d27de0457575c9bb581965c41c854505806c011546658 WatchSource:0}: Error finding container 2db9420c2bfd094fbd3d27de0457575c9bb581965c41c854505806c011546658: Status 404 returned error can't find the container with id 2db9420c2bfd094fbd3d27de0457575c9bb581965c41c854505806c011546658 Nov 01 06:45:01.198255 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:45:01.198219 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c4bca9d_4e46_44e4_9270_ea42ada707bb.slice/crio-3a875f927888c1b88987dc61c033fdc848296fa235dedf48277b9c10b65545d2 WatchSource:0}: Error finding container 3a875f927888c1b88987dc61c033fdc848296fa235dedf48277b9c10b65545d2: Status 404 returned error can't find the container with id 3a875f927888c1b88987dc61c033fdc848296fa235dedf48277b9c10b65545d2 Nov 01 06:45:01.402109 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:01.401993 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56" Nov 01 06:45:01.416873 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:01.416848 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56" Nov 01 06:45:02.046932 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:02.046897 5040 generic.go:356] "Generic (PLEG): container finished" podID="1a8d4067-a520-4890-9d89-e73e382438b1" containerID="579e5259deb7ab265bad60511f1977e5b29cf1d04f03db124be040da54636c54" exitCode=60 Nov 01 06:45:02.047096 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:02.046982 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891925-zh28x" event={"ID":"1a8d4067-a520-4890-9d89-e73e382438b1","Type":"ContainerDied","Data":"579e5259deb7ab265bad60511f1977e5b29cf1d04f03db124be040da54636c54"} Nov 01 06:45:02.047302 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:02.047285 5040 scope.go:122] "RemoveContainer" containerID="579e5259deb7ab265bad60511f1977e5b29cf1d04f03db124be040da54636c54" Nov 01 06:45:02.048949 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:02.048791 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891925-c7wn9" event={"ID":"d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9","Type":"ContainerStarted","Data":"6d1e41ce631ac63006c30a2afab9e22eeef48f82e559ef957cfe64f78c87202e"} Nov 01 06:45:02.048949 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:02.048834 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891925-c7wn9" event={"ID":"d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9","Type":"ContainerStarted","Data":"dfbda5c38d4d60ecf127ea158709e0eb591bf7ac2c2aa4ccb1233bb40349c838"} Nov 01 06:45:02.050566 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:02.050540 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891925-5r4hb" event={"ID":"0431f2e8-f3aa-4f41-9360-0f12c5baa029","Type":"ContainerStarted","Data":"e28578024f0bae4069d026e41ee0125b1fbb51c0f5ebfa2c537bcdf71d5975ca"} Nov 01 06:45:02.050621 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:02.050580 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891925-5r4hb" event={"ID":"0431f2e8-f3aa-4f41-9360-0f12c5baa029","Type":"ContainerStarted","Data":"79248018f278fb272d67eb51bc230899c99a7cc21226665a3ed5fe6732370f5c"} Nov 01 06:45:02.052361 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:02.052338 5040 generic.go:356] "Generic (PLEG): container finished" podID="c57c77c0-2554-4f46-9c1a-a8321a993953" containerID="d108061fc00a0aac7f3864b2925aba9a42d8b6b61d2642baa3644026a685a603" exitCode=60 Nov 01 06:45:02.052464 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:02.052423 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891925-chf96" event={"ID":"c57c77c0-2554-4f46-9c1a-a8321a993953","Type":"ContainerDied","Data":"d108061fc00a0aac7f3864b2925aba9a42d8b6b61d2642baa3644026a685a603"} Nov 01 06:45:02.052678 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:02.052657 5040 scope.go:122] "RemoveContainer" containerID="d108061fc00a0aac7f3864b2925aba9a42d8b6b61d2642baa3644026a685a603" Nov 01 06:45:02.054386 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:02.054364 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891925-dqkfq" event={"ID":"7482f875-ca8d-4b85-ab43-5e7eea31efca","Type":"ContainerStarted","Data":"af60b7913a7a4f828820ee24fc9a9d3c9b2cb48f1842b6cdb479b586e35a72fa"} Nov 01 06:45:02.054518 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:02.054419 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891925-dqkfq" event={"ID":"7482f875-ca8d-4b85-ab43-5e7eea31efca","Type":"ContainerStarted","Data":"2db9420c2bfd094fbd3d27de0457575c9bb581965c41c854505806c011546658"} Nov 01 06:45:02.056014 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:02.055948 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891925-cx2hq" event={"ID":"1c4bca9d-4e46-44e4-9270-ea42ada707bb","Type":"ContainerStarted","Data":"bd9757caeff8ce4e4e0e5b7e07e3833e1154765e7d4560c3ebe219d489858409"} Nov 01 06:45:02.056014 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:02.055992 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891925-cx2hq" event={"ID":"1c4bca9d-4e46-44e4-9270-ea42ada707bb","Type":"ContainerStarted","Data":"3a875f927888c1b88987dc61c033fdc848296fa235dedf48277b9c10b65545d2"} Nov 01 06:45:02.057559 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:02.057538 5040 generic.go:356] "Generic (PLEG): container finished" podID="e328a3ce-7f5d-4599-a616-b49f6f18a510" containerID="2daa9db86b54ac5727d4ab782e15eb0f244b00a768221d6ffb959b57694d150f" exitCode=60 Nov 01 06:45:02.057657 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:02.057617 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-747b7b-29891925-g8kd9" event={"ID":"e328a3ce-7f5d-4599-a616-b49f6f18a510","Type":"ContainerDied","Data":"2daa9db86b54ac5727d4ab782e15eb0f244b00a768221d6ffb959b57694d150f"} Nov 01 06:45:02.057819 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:02.057799 5040 scope.go:122] "RemoveContainer" containerID="2daa9db86b54ac5727d4ab782e15eb0f244b00a768221d6ffb959b57694d150f" Nov 01 06:45:02.059907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:02.059887 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5be4fa-29891925-l24ff" event={"ID":"4957cc07-721f-40ca-85e7-3d9ca330096e","Type":"ContainerStarted","Data":"06e6716ce6b724064c63bf238f0b928ab3a0cdbf9d00bbbf01ab37db2a1d0b30"} Nov 01 06:45:02.060011 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:02.059930 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5be4fa-29891925-l24ff" event={"ID":"4957cc07-721f-40ca-85e7-3d9ca330096e","Type":"ContainerStarted","Data":"f9bcb225d331f300f6a0b21ba748b7d46a3764de91e8195dd2b0f08cf172e329"} Nov 01 06:45:02.073105 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:02.073073 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56" Nov 01 06:45:02.514360 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:02.514306 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891925-cx2hq" podStartSLOduration=2.5142885489999998 podStartE2EDuration="2.514288549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:45:02.513671292 +0000 UTC m=+3581.168788461" watchObservedRunningTime="2026-11-01 06:45:02.514288549 +0000 UTC m=+3581.169405714" Nov 01 06:45:02.630045 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:02.629987 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891925-5r4hb" podStartSLOduration=2.629972223 podStartE2EDuration="2.629972223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:45:02.628978243 +0000 UTC m=+3581.284095411" watchObservedRunningTime="2026-11-01 06:45:02.629972223 +0000 UTC m=+3581.285089390" Nov 01 06:45:02.861803 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:02.861751 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891925-dqkfq" podStartSLOduration=2.861739079 podStartE2EDuration="2.861739079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:45:02.859852309 +0000 UTC m=+3581.514969478" watchObservedRunningTime="2026-11-01 06:45:02.861739079 +0000 UTC m=+3581.516856245" Nov 01 06:45:02.979041 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:02.978931 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5be4fa-29891925-l24ff" podStartSLOduration=2.978917237 podStartE2EDuration="2.978917237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:45:02.97583358 +0000 UTC m=+3581.630950747" watchObservedRunningTime="2026-11-01 06:45:02.978917237 +0000 UTC m=+3581.634034404" Nov 01 06:45:03.068290 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:03.068251 5040 generic.go:356] "Generic (PLEG): container finished" podID="d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9" containerID="6d1e41ce631ac63006c30a2afab9e22eeef48f82e559ef957cfe64f78c87202e" exitCode=60 Nov 01 06:45:03.068516 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:03.068341 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891925-c7wn9" event={"ID":"d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9","Type":"ContainerDied","Data":"6d1e41ce631ac63006c30a2afab9e22eeef48f82e559ef957cfe64f78c87202e"} Nov 01 06:45:03.068616 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:03.068601 5040 scope.go:122] "RemoveContainer" containerID="6d1e41ce631ac63006c30a2afab9e22eeef48f82e559ef957cfe64f78c87202e" Nov 01 06:45:03.070394 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:03.070369 5040 generic.go:356] "Generic (PLEG): container finished" podID="0431f2e8-f3aa-4f41-9360-0f12c5baa029" containerID="e28578024f0bae4069d026e41ee0125b1fbb51c0f5ebfa2c537bcdf71d5975ca" exitCode=60 Nov 01 06:45:03.070555 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:03.070441 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891925-5r4hb" event={"ID":"0431f2e8-f3aa-4f41-9360-0f12c5baa029","Type":"ContainerDied","Data":"e28578024f0bae4069d026e41ee0125b1fbb51c0f5ebfa2c537bcdf71d5975ca"} Nov 01 06:45:03.070678 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:03.070663 5040 scope.go:122] "RemoveContainer" containerID="e28578024f0bae4069d026e41ee0125b1fbb51c0f5ebfa2c537bcdf71d5975ca" Nov 01 06:45:03.072598 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:03.072578 5040 generic.go:356] "Generic (PLEG): container finished" podID="c57c77c0-2554-4f46-9c1a-a8321a993953" containerID="4d8bdd9c7ce4b8c1b117866585d7ac5f7928eb51ec64c6c8ef8729e48e02542e" exitCode=60 Nov 01 06:45:03.072713 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:03.072664 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891925-chf96" event={"ID":"c57c77c0-2554-4f46-9c1a-a8321a993953","Type":"ContainerDied","Data":"4d8bdd9c7ce4b8c1b117866585d7ac5f7928eb51ec64c6c8ef8729e48e02542e"} Nov 01 06:45:03.072775 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:03.072719 5040 scope.go:122] "RemoveContainer" containerID="d108061fc00a0aac7f3864b2925aba9a42d8b6b61d2642baa3644026a685a603" Nov 01 06:45:03.073073 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:03.072870 5040 scope.go:122] "RemoveContainer" containerID="4d8bdd9c7ce4b8c1b117866585d7ac5f7928eb51ec64c6c8ef8729e48e02542e" Nov 01 06:45:03.073563 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:45:03.073539 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891925-chf96_odh-ai-gateway-infra(c57c77c0-2554-4f46-9c1a-a8321a993953)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891925-chf96" podUID="c57c77c0-2554-4f46-9c1a-a8321a993953" Nov 01 06:45:03.074686 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:03.074656 5040 generic.go:356] "Generic (PLEG): container finished" podID="7482f875-ca8d-4b85-ab43-5e7eea31efca" containerID="af60b7913a7a4f828820ee24fc9a9d3c9b2cb48f1842b6cdb479b586e35a72fa" exitCode=60 Nov 01 06:45:03.074767 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:03.074732 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891925-dqkfq" event={"ID":"7482f875-ca8d-4b85-ab43-5e7eea31efca","Type":"ContainerDied","Data":"af60b7913a7a4f828820ee24fc9a9d3c9b2cb48f1842b6cdb479b586e35a72fa"} Nov 01 06:45:03.074915 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:03.074899 5040 scope.go:122] "RemoveContainer" containerID="af60b7913a7a4f828820ee24fc9a9d3c9b2cb48f1842b6cdb479b586e35a72fa" Nov 01 06:45:03.076594 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:03.076570 5040 generic.go:356] "Generic (PLEG): container finished" podID="1c4bca9d-4e46-44e4-9270-ea42ada707bb" containerID="bd9757caeff8ce4e4e0e5b7e07e3833e1154765e7d4560c3ebe219d489858409" exitCode=60 Nov 01 06:45:03.076695 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:03.076616 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891925-cx2hq" event={"ID":"1c4bca9d-4e46-44e4-9270-ea42ada707bb","Type":"ContainerDied","Data":"bd9757caeff8ce4e4e0e5b7e07e3833e1154765e7d4560c3ebe219d489858409"} Nov 01 06:45:03.076854 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:03.076839 5040 scope.go:122] "RemoveContainer" containerID="bd9757caeff8ce4e4e0e5b7e07e3833e1154765e7d4560c3ebe219d489858409" Nov 01 06:45:03.078693 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:03.078674 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-747b7b-29891925-g8kd9" event={"ID":"e328a3ce-7f5d-4599-a616-b49f6f18a510","Type":"ContainerStarted","Data":"5cc340120a879508fa766f7c21fea16812f4f7b2f9ab43a0ea73f268a9287746"} Nov 01 06:45:03.081582 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:03.081560 5040 generic.go:356] "Generic (PLEG): container finished" podID="4957cc07-721f-40ca-85e7-3d9ca330096e" containerID="06e6716ce6b724064c63bf238f0b928ab3a0cdbf9d00bbbf01ab37db2a1d0b30" exitCode=7 Nov 01 06:45:03.081972 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:03.081928 5040 scope.go:122] "RemoveContainer" containerID="06e6716ce6b724064c63bf238f0b928ab3a0cdbf9d00bbbf01ab37db2a1d0b30" Nov 01 06:45:03.081972 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:03.081954 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5be4fa-29891925-l24ff" event={"ID":"4957cc07-721f-40ca-85e7-3d9ca330096e","Type":"ContainerDied","Data":"06e6716ce6b724064c63bf238f0b928ab3a0cdbf9d00bbbf01ab37db2a1d0b30"} Nov 01 06:45:03.084386 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:03.084359 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891925-zh28x" event={"ID":"1a8d4067-a520-4890-9d89-e73e382438b1","Type":"ContainerStarted","Data":"9c901f50add4cc2d72d87f9dedbabad7cab95004737a211de9c0aefeef981740"} Nov 01 06:45:03.096154 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:03.093754 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891925-c7wn9" podStartSLOduration=3.093736861 podStartE2EDuration="3.093736861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:45:03.092305621 +0000 UTC m=+3581.747422790" watchObservedRunningTime="2026-11-01 06:45:03.093736861 +0000 UTC m=+3581.748854028" Nov 01 06:45:03.558234 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:03.558144 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-747b7b-29891925-g8kd9" podStartSLOduration=3.5581308849999997 podStartE2EDuration="3.558130885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:45:03.556215319 +0000 UTC m=+3582.211332489" watchObservedRunningTime="2026-11-01 06:45:03.558130885 +0000 UTC m=+3582.213248052" Nov 01 06:45:03.677314 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:03.677259 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891925-zh28x" podStartSLOduration=3.6772445559999998 podStartE2EDuration="3.677244556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:45:03.675970218 +0000 UTC m=+3582.331087385" watchObservedRunningTime="2026-11-01 06:45:03.677244556 +0000 UTC m=+3582.332361724" Nov 01 06:45:04.092544 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:04.092509 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891925-c7wn9" event={"ID":"d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9","Type":"ContainerStarted","Data":"aaebd700e48405b2ff03806b19926d13fcc79b98f1f0fca762773006721b317f"} Nov 01 06:45:04.094935 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:04.094905 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891925-5r4hb" event={"ID":"0431f2e8-f3aa-4f41-9360-0f12c5baa029","Type":"ContainerStarted","Data":"e45655ebd3e2926d49986099f3da97f2efe13f010fbb38c797b8106c6ab40673"} Nov 01 06:45:04.096814 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:04.096788 5040 scope.go:122] "RemoveContainer" containerID="4d8bdd9c7ce4b8c1b117866585d7ac5f7928eb51ec64c6c8ef8729e48e02542e" Nov 01 06:45:04.097057 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:45:04.097031 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891925-chf96_odh-ai-gateway-infra(c57c77c0-2554-4f46-9c1a-a8321a993953)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891925-chf96" podUID="c57c77c0-2554-4f46-9c1a-a8321a993953" Nov 01 06:45:04.099779 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:04.099723 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891925-dqkfq" event={"ID":"7482f875-ca8d-4b85-ab43-5e7eea31efca","Type":"ContainerStarted","Data":"7452b06fa7fc7fcda3af6653df1a6b37d90dac4014494280086bfaab5c530439"} Nov 01 06:45:04.102899 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:04.102597 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891925-cx2hq" event={"ID":"1c4bca9d-4e46-44e4-9270-ea42ada707bb","Type":"ContainerStarted","Data":"923e48b5e562b968fd19d9ccbc00f7e5de47eacae4fb5d8e5d732aa426524827"} Nov 01 06:45:04.104901 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:04.104873 5040 generic.go:356] "Generic (PLEG): container finished" podID="e328a3ce-7f5d-4599-a616-b49f6f18a510" containerID="5cc340120a879508fa766f7c21fea16812f4f7b2f9ab43a0ea73f268a9287746" exitCode=60 Nov 01 06:45:04.104992 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:04.104959 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-747b7b-29891925-g8kd9" event={"ID":"e328a3ce-7f5d-4599-a616-b49f6f18a510","Type":"ContainerDied","Data":"5cc340120a879508fa766f7c21fea16812f4f7b2f9ab43a0ea73f268a9287746"} Nov 01 06:45:04.105031 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:04.105006 5040 scope.go:122] "RemoveContainer" containerID="2daa9db86b54ac5727d4ab782e15eb0f244b00a768221d6ffb959b57694d150f" Nov 01 06:45:04.105376 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:04.105358 5040 scope.go:122] "RemoveContainer" containerID="5cc340120a879508fa766f7c21fea16812f4f7b2f9ab43a0ea73f268a9287746" Nov 01 06:45:04.105676 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:45:04.105645 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-747b7b-29891925-g8kd9_odh-ai-gateway-infra(e328a3ce-7f5d-4599-a616-b49f6f18a510)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-747b7b-29891925-g8kd9" podUID="e328a3ce-7f5d-4599-a616-b49f6f18a510" Nov 01 06:45:04.108127 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:04.108107 5040 generic.go:356] "Generic (PLEG): container finished" podID="4957cc07-721f-40ca-85e7-3d9ca330096e" containerID="c5c4976b59d16061e9fc3ecf5b8f541886272d810d3b1794e8a6b2764f5c1d1e" exitCode=7 Nov 01 06:45:04.108284 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:04.108190 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5be4fa-29891925-l24ff" event={"ID":"4957cc07-721f-40ca-85e7-3d9ca330096e","Type":"ContainerDied","Data":"c5c4976b59d16061e9fc3ecf5b8f541886272d810d3b1794e8a6b2764f5c1d1e"} Nov 01 06:45:04.108407 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:04.108392 5040 scope.go:122] "RemoveContainer" containerID="c5c4976b59d16061e9fc3ecf5b8f541886272d810d3b1794e8a6b2764f5c1d1e" Nov 01 06:45:04.108612 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:45:04.108591 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-5be4fa-29891925-l24ff_odh-ai-gateway-infra(4957cc07-721f-40ca-85e7-3d9ca330096e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5be4fa-29891925-l24ff" podUID="4957cc07-721f-40ca-85e7-3d9ca330096e" Nov 01 06:45:04.110154 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:04.110131 5040 generic.go:356] "Generic (PLEG): container finished" podID="1a8d4067-a520-4890-9d89-e73e382438b1" containerID="9c901f50add4cc2d72d87f9dedbabad7cab95004737a211de9c0aefeef981740" exitCode=60 Nov 01 06:45:04.110285 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:04.110220 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891925-zh28x" event={"ID":"1a8d4067-a520-4890-9d89-e73e382438b1","Type":"ContainerDied","Data":"9c901f50add4cc2d72d87f9dedbabad7cab95004737a211de9c0aefeef981740"} Nov 01 06:45:04.110548 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:04.110468 5040 scope.go:122] "RemoveContainer" containerID="9c901f50add4cc2d72d87f9dedbabad7cab95004737a211de9c0aefeef981740" Nov 01 06:45:04.110744 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:45:04.110726 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891925-zh28x_odh-ai-gateway-infra(1a8d4067-a520-4890-9d89-e73e382438b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891925-zh28x" podUID="1a8d4067-a520-4890-9d89-e73e382438b1" Nov 01 06:45:04.295956 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:04.295924 5040 scope.go:122] "RemoveContainer" containerID="06e6716ce6b724064c63bf238f0b928ab3a0cdbf9d00bbbf01ab37db2a1d0b30" Nov 01 06:45:04.309263 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:04.309243 5040 scope.go:122] "RemoveContainer" containerID="579e5259deb7ab265bad60511f1977e5b29cf1d04f03db124be040da54636c54" Nov 01 06:45:05.121008 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:05.120977 5040 scope.go:122] "RemoveContainer" containerID="5cc340120a879508fa766f7c21fea16812f4f7b2f9ab43a0ea73f268a9287746" Nov 01 06:45:05.121409 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:45:05.121264 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-747b7b-29891925-g8kd9_odh-ai-gateway-infra(e328a3ce-7f5d-4599-a616-b49f6f18a510)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-747b7b-29891925-g8kd9" podUID="e328a3ce-7f5d-4599-a616-b49f6f18a510" Nov 01 06:45:05.122532 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:05.122509 5040 scope.go:122] "RemoveContainer" containerID="c5c4976b59d16061e9fc3ecf5b8f541886272d810d3b1794e8a6b2764f5c1d1e" Nov 01 06:45:05.122777 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:45:05.122754 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-5be4fa-29891925-l24ff_odh-ai-gateway-infra(4957cc07-721f-40ca-85e7-3d9ca330096e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5be4fa-29891925-l24ff" podUID="4957cc07-721f-40ca-85e7-3d9ca330096e" Nov 01 06:45:05.124161 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:05.124141 5040 scope.go:122] "RemoveContainer" containerID="9c901f50add4cc2d72d87f9dedbabad7cab95004737a211de9c0aefeef981740" Nov 01 06:45:05.124375 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:45:05.124356 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891925-zh28x_odh-ai-gateway-infra(1a8d4067-a520-4890-9d89-e73e382438b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891925-zh28x" podUID="1a8d4067-a520-4890-9d89-e73e382438b1" Nov 01 06:45:05.126089 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:05.126064 5040 generic.go:356] "Generic (PLEG): container finished" podID="d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9" containerID="aaebd700e48405b2ff03806b19926d13fcc79b98f1f0fca762773006721b317f" exitCode=60 Nov 01 06:45:05.126209 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:05.126143 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891925-c7wn9" event={"ID":"d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9","Type":"ContainerDied","Data":"aaebd700e48405b2ff03806b19926d13fcc79b98f1f0fca762773006721b317f"} Nov 01 06:45:05.126276 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:05.126207 5040 scope.go:122] "RemoveContainer" containerID="6d1e41ce631ac63006c30a2afab9e22eeef48f82e559ef957cfe64f78c87202e" Nov 01 06:45:05.126392 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:05.126376 5040 scope.go:122] "RemoveContainer" containerID="aaebd700e48405b2ff03806b19926d13fcc79b98f1f0fca762773006721b317f" Nov 01 06:45:05.126671 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:45:05.126636 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891925-c7wn9_odh-ai-gateway-infra(d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891925-c7wn9" podUID="d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9" Nov 01 06:45:05.128984 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:05.128965 5040 generic.go:356] "Generic (PLEG): container finished" podID="0431f2e8-f3aa-4f41-9360-0f12c5baa029" containerID="e45655ebd3e2926d49986099f3da97f2efe13f010fbb38c797b8106c6ab40673" exitCode=60 Nov 01 06:45:05.129093 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:05.129045 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891925-5r4hb" event={"ID":"0431f2e8-f3aa-4f41-9360-0f12c5baa029","Type":"ContainerDied","Data":"e45655ebd3e2926d49986099f3da97f2efe13f010fbb38c797b8106c6ab40673"} Nov 01 06:45:05.129230 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:05.129211 5040 scope.go:122] "RemoveContainer" containerID="e45655ebd3e2926d49986099f3da97f2efe13f010fbb38c797b8106c6ab40673" Nov 01 06:45:05.129420 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:45:05.129397 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-cfe368-29891925-5r4hb_odh-ai-gateway-infra(0431f2e8-f3aa-4f41-9360-0f12c5baa029)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891925-5r4hb" podUID="0431f2e8-f3aa-4f41-9360-0f12c5baa029" Nov 01 06:45:05.130986 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:05.130961 5040 generic.go:356] "Generic (PLEG): container finished" podID="7482f875-ca8d-4b85-ab43-5e7eea31efca" containerID="7452b06fa7fc7fcda3af6653df1a6b37d90dac4014494280086bfaab5c530439" exitCode=60 Nov 01 06:45:05.131072 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:05.131040 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891925-dqkfq" event={"ID":"7482f875-ca8d-4b85-ab43-5e7eea31efca","Type":"ContainerDied","Data":"7452b06fa7fc7fcda3af6653df1a6b37d90dac4014494280086bfaab5c530439"} Nov 01 06:45:05.131207 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:05.131193 5040 scope.go:122] "RemoveContainer" containerID="7452b06fa7fc7fcda3af6653df1a6b37d90dac4014494280086bfaab5c530439" Nov 01 06:45:05.131408 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:45:05.131387 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-3986bf-29891925-dqkfq_odh-ai-gateway-infra(7482f875-ca8d-4b85-ab43-5e7eea31efca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891925-dqkfq" podUID="7482f875-ca8d-4b85-ab43-5e7eea31efca" Nov 01 06:45:05.132929 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:05.132907 5040 generic.go:356] "Generic (PLEG): container finished" podID="1c4bca9d-4e46-44e4-9270-ea42ada707bb" containerID="923e48b5e562b968fd19d9ccbc00f7e5de47eacae4fb5d8e5d732aa426524827" exitCode=60 Nov 01 06:45:05.133010 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:05.132937 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891925-cx2hq" event={"ID":"1c4bca9d-4e46-44e4-9270-ea42ada707bb","Type":"ContainerDied","Data":"923e48b5e562b968fd19d9ccbc00f7e5de47eacae4fb5d8e5d732aa426524827"} Nov 01 06:45:05.133123 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:05.133109 5040 scope.go:122] "RemoveContainer" containerID="923e48b5e562b968fd19d9ccbc00f7e5de47eacae4fb5d8e5d732aa426524827" Nov 01 06:45:05.133295 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:45:05.133278 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891925-cx2hq_odh-ai-gateway-infra(1c4bca9d-4e46-44e4-9270-ea42ada707bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891925-cx2hq" podUID="1c4bca9d-4e46-44e4-9270-ea42ada707bb" Nov 01 06:45:05.140629 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:05.140610 5040 scope.go:122] "RemoveContainer" containerID="e28578024f0bae4069d026e41ee0125b1fbb51c0f5ebfa2c537bcdf71d5975ca" Nov 01 06:45:05.153357 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:05.153341 5040 scope.go:122] "RemoveContainer" containerID="af60b7913a7a4f828820ee24fc9a9d3c9b2cb48f1842b6cdb479b586e35a72fa" Nov 01 06:45:05.165814 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:05.165789 5040 scope.go:122] "RemoveContainer" containerID="bd9757caeff8ce4e4e0e5b7e07e3833e1154765e7d4560c3ebe219d489858409" Nov 01 06:45:06.047269 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:06.047236 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76d2a29e-xrmw7"] Nov 01 06:45:06.050865 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:06.050832 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76d2a29e-xrmw7"] Nov 01 06:45:06.141812 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:06.141770 5040 scope.go:122] "RemoveContainer" containerID="aaebd700e48405b2ff03806b19926d13fcc79b98f1f0fca762773006721b317f" Nov 01 06:45:06.142264 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:45:06.142057 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891925-c7wn9_odh-ai-gateway-infra(d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891925-c7wn9" podUID="d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9" Nov 01 06:45:06.143524 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:06.143503 5040 scope.go:122] "RemoveContainer" containerID="e45655ebd3e2926d49986099f3da97f2efe13f010fbb38c797b8106c6ab40673" Nov 01 06:45:06.143741 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:45:06.143720 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-cfe368-29891925-5r4hb_odh-ai-gateway-infra(0431f2e8-f3aa-4f41-9360-0f12c5baa029)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891925-5r4hb" podUID="0431f2e8-f3aa-4f41-9360-0f12c5baa029" Nov 01 06:45:06.145121 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:06.145104 5040 scope.go:122] "RemoveContainer" containerID="7452b06fa7fc7fcda3af6653df1a6b37d90dac4014494280086bfaab5c530439" Nov 01 06:45:06.145339 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:45:06.145318 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-3986bf-29891925-dqkfq_odh-ai-gateway-infra(7482f875-ca8d-4b85-ab43-5e7eea31efca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891925-dqkfq" podUID="7482f875-ca8d-4b85-ab43-5e7eea31efca" Nov 01 06:45:06.146611 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:06.146592 5040 scope.go:122] "RemoveContainer" containerID="923e48b5e562b968fd19d9ccbc00f7e5de47eacae4fb5d8e5d732aa426524827" Nov 01 06:45:06.146833 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:45:06.146813 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891925-cx2hq_odh-ai-gateway-infra(1c4bca9d-4e46-44e4-9270-ea42ada707bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891925-cx2hq" podUID="1c4bca9d-4e46-44e4-9270-ea42ada707bb" Nov 01 06:45:07.515007 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:07.514964 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2838d17e-42d0-4388-9e7b-8075d9b9503d" path="/var/lib/kubelet/pods/2838d17e-42d0-4388-9e7b-8075d9b9503d/volumes" Nov 01 06:45:09.973922 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:45:09.973879 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:45:10.048400 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:10.048363 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88b760ca-lfph6"] Nov 01 06:45:10.098731 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:10.098696 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88b760ca-lfph6"] Nov 01 06:45:11.550505 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:11.540891 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d238dea-5ba1-4080-a7d9-74a985c08e99" path="/var/lib/kubelet/pods/0d238dea-5ba1-4080-a7d9-74a985c08e99/volumes" Nov 01 06:45:14.519321 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:14.519284 5040 scope.go:122] "RemoveContainer" containerID="923e48b5e562b968fd19d9ccbc00f7e5de47eacae4fb5d8e5d732aa426524827" Nov 01 06:45:14.519886 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:14.519442 5040 scope.go:122] "RemoveContainer" containerID="4d8bdd9c7ce4b8c1b117866585d7ac5f7928eb51ec64c6c8ef8729e48e02542e" Nov 01 06:45:15.232572 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:15.232532 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891925-chf96" event={"ID":"c57c77c0-2554-4f46-9c1a-a8321a993953","Type":"ContainerStarted","Data":"0f2ce942a773ed0d1085577e333d9914a244a2b3902953fbcf3c69fd37452951"} Nov 01 06:45:15.234612 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:15.234584 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891925-cx2hq" event={"ID":"1c4bca9d-4e46-44e4-9270-ea42ada707bb","Type":"ContainerStarted","Data":"159100641f19e70b6b7d8cb9725113eeb211c2e387b4b6cfe68d34e8813781bd"} Nov 01 06:45:15.350287 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:15.350183 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891925-chf96" podStartSLOduration=15.350169321 podStartE2EDuration="15.350169321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:45:15.348397053 +0000 UTC m=+3594.003514219" watchObservedRunningTime="2026-11-01 06:45:15.350169321 +0000 UTC m=+3594.005286489" Nov 01 06:45:15.514510 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:15.514182 5040 scope.go:122] "RemoveContainer" containerID="e45655ebd3e2926d49986099f3da97f2efe13f010fbb38c797b8106c6ab40673" Nov 01 06:45:15.518527 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:15.515076 5040 scope.go:122] "RemoveContainer" containerID="9c901f50add4cc2d72d87f9dedbabad7cab95004737a211de9c0aefeef981740" Nov 01 06:45:15.519619 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:15.519229 5040 scope.go:122] "RemoveContainer" containerID="7452b06fa7fc7fcda3af6653df1a6b37d90dac4014494280086bfaab5c530439" Nov 01 06:45:16.243758 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:16.243724 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891925-zh28x" event={"ID":"1a8d4067-a520-4890-9d89-e73e382438b1","Type":"ContainerStarted","Data":"10928088b49718c4eb39a880a1d0b7ef97dfe8ffe6931c48ea58ac6ad456bb90"} Nov 01 06:45:16.245748 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:16.245716 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891925-5r4hb" event={"ID":"0431f2e8-f3aa-4f41-9360-0f12c5baa029","Type":"ContainerStarted","Data":"08152e853ef13fc17877f5ed433b2bd0d5602b8cd35cff3e4664289d6d064068"} Nov 01 06:45:16.247442 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:16.247421 5040 generic.go:356] "Generic (PLEG): container finished" podID="c57c77c0-2554-4f46-9c1a-a8321a993953" containerID="0f2ce942a773ed0d1085577e333d9914a244a2b3902953fbcf3c69fd37452951" exitCode=60 Nov 01 06:45:16.247572 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:16.247507 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891925-chf96" event={"ID":"c57c77c0-2554-4f46-9c1a-a8321a993953","Type":"ContainerDied","Data":"0f2ce942a773ed0d1085577e333d9914a244a2b3902953fbcf3c69fd37452951"} Nov 01 06:45:16.247572 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:16.247555 5040 scope.go:122] "RemoveContainer" containerID="4d8bdd9c7ce4b8c1b117866585d7ac5f7928eb51ec64c6c8ef8729e48e02542e" Nov 01 06:45:16.247841 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:16.247816 5040 scope.go:122] "RemoveContainer" containerID="0f2ce942a773ed0d1085577e333d9914a244a2b3902953fbcf3c69fd37452951" Nov 01 06:45:16.248122 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:45:16.248094 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891925-chf96_odh-ai-gateway-infra(c57c77c0-2554-4f46-9c1a-a8321a993953)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891925-chf96" podUID="c57c77c0-2554-4f46-9c1a-a8321a993953" Nov 01 06:45:16.249444 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:16.249422 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891925-dqkfq" event={"ID":"7482f875-ca8d-4b85-ab43-5e7eea31efca","Type":"ContainerStarted","Data":"8b4c8602821104a48a2c23ef4610f5b4563f011f2ceafb37d8f0853eade6cea8"} Nov 01 06:45:16.251206 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:16.251186 5040 generic.go:356] "Generic (PLEG): container finished" podID="1c4bca9d-4e46-44e4-9270-ea42ada707bb" containerID="159100641f19e70b6b7d8cb9725113eeb211c2e387b4b6cfe68d34e8813781bd" exitCode=60 Nov 01 06:45:16.251281 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:16.251227 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891925-cx2hq" event={"ID":"1c4bca9d-4e46-44e4-9270-ea42ada707bb","Type":"ContainerDied","Data":"159100641f19e70b6b7d8cb9725113eeb211c2e387b4b6cfe68d34e8813781bd"} Nov 01 06:45:16.251474 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:16.251460 5040 scope.go:122] "RemoveContainer" containerID="159100641f19e70b6b7d8cb9725113eeb211c2e387b4b6cfe68d34e8813781bd" Nov 01 06:45:16.251656 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:45:16.251639 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891925-cx2hq_odh-ai-gateway-infra(1c4bca9d-4e46-44e4-9270-ea42ada707bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891925-cx2hq" podUID="1c4bca9d-4e46-44e4-9270-ea42ada707bb" Nov 01 06:45:16.260850 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:16.260834 5040 scope.go:122] "RemoveContainer" containerID="923e48b5e562b968fd19d9ccbc00f7e5de47eacae4fb5d8e5d732aa426524827" Nov 01 06:45:16.361145 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:16.361113 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891925-chf96"] Nov 01 06:45:16.485952 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:16.485916 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891925-cx2hq"] Nov 01 06:45:16.522505 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:16.509250 5040 scope.go:122] "RemoveContainer" containerID="c5c4976b59d16061e9fc3ecf5b8f541886272d810d3b1794e8a6b2764f5c1d1e" Nov 01 06:45:16.523885 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:16.523806 5040 scope.go:122] "RemoveContainer" containerID="5cc340120a879508fa766f7c21fea16812f4f7b2f9ab43a0ea73f268a9287746" Nov 01 06:45:17.261032 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:17.260993 5040 generic.go:356] "Generic (PLEG): container finished" podID="1a8d4067-a520-4890-9d89-e73e382438b1" containerID="10928088b49718c4eb39a880a1d0b7ef97dfe8ffe6931c48ea58ac6ad456bb90" exitCode=60 Nov 01 06:45:17.261207 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:17.261070 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891925-zh28x" event={"ID":"1a8d4067-a520-4890-9d89-e73e382438b1","Type":"ContainerDied","Data":"10928088b49718c4eb39a880a1d0b7ef97dfe8ffe6931c48ea58ac6ad456bb90"} Nov 01 06:45:17.261207 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:17.261147 5040 scope.go:122] "RemoveContainer" containerID="9c901f50add4cc2d72d87f9dedbabad7cab95004737a211de9c0aefeef981740" Nov 01 06:45:17.261647 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:17.261625 5040 scope.go:122] "RemoveContainer" containerID="10928088b49718c4eb39a880a1d0b7ef97dfe8ffe6931c48ea58ac6ad456bb90" Nov 01 06:45:17.261996 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:45:17.261964 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891925-zh28x_odh-ai-gateway-infra(1a8d4067-a520-4890-9d89-e73e382438b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891925-zh28x" podUID="1a8d4067-a520-4890-9d89-e73e382438b1" Nov 01 06:45:17.264019 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:17.263993 5040 generic.go:356] "Generic (PLEG): container finished" podID="0431f2e8-f3aa-4f41-9360-0f12c5baa029" containerID="08152e853ef13fc17877f5ed433b2bd0d5602b8cd35cff3e4664289d6d064068" exitCode=60 Nov 01 06:45:17.264134 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:17.264068 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891925-5r4hb" event={"ID":"0431f2e8-f3aa-4f41-9360-0f12c5baa029","Type":"ContainerDied","Data":"08152e853ef13fc17877f5ed433b2bd0d5602b8cd35cff3e4664289d6d064068"} Nov 01 06:45:17.264596 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:17.264567 5040 scope.go:122] "RemoveContainer" containerID="08152e853ef13fc17877f5ed433b2bd0d5602b8cd35cff3e4664289d6d064068" Nov 01 06:45:17.264924 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:45:17.264897 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-cfe368-29891925-5r4hb_odh-ai-gateway-infra(0431f2e8-f3aa-4f41-9360-0f12c5baa029)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891925-5r4hb" podUID="0431f2e8-f3aa-4f41-9360-0f12c5baa029" Nov 01 06:45:17.270679 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:17.270658 5040 generic.go:356] "Generic (PLEG): container finished" podID="7482f875-ca8d-4b85-ab43-5e7eea31efca" containerID="8b4c8602821104a48a2c23ef4610f5b4563f011f2ceafb37d8f0853eade6cea8" exitCode=60 Nov 01 06:45:17.270784 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:17.270728 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891925-dqkfq" event={"ID":"7482f875-ca8d-4b85-ab43-5e7eea31efca","Type":"ContainerDied","Data":"8b4c8602821104a48a2c23ef4610f5b4563f011f2ceafb37d8f0853eade6cea8"} Nov 01 06:45:17.271052 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:17.271030 5040 scope.go:122] "RemoveContainer" containerID="8b4c8602821104a48a2c23ef4610f5b4563f011f2ceafb37d8f0853eade6cea8" Nov 01 06:45:17.271295 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:45:17.271264 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-3986bf-29891925-dqkfq_odh-ai-gateway-infra(7482f875-ca8d-4b85-ab43-5e7eea31efca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891925-dqkfq" podUID="7482f875-ca8d-4b85-ab43-5e7eea31efca" Nov 01 06:45:17.275522 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:17.274884 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-747b7b-29891925-g8kd9" event={"ID":"e328a3ce-7f5d-4599-a616-b49f6f18a510","Type":"ContainerStarted","Data":"ca35cb5378aeda40df800668fc34c39f1f444978614dcd76d2783ec03c0e474e"} Nov 01 06:45:17.279690 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:17.279657 5040 scope.go:122] "RemoveContainer" containerID="e45655ebd3e2926d49986099f3da97f2efe13f010fbb38c797b8106c6ab40673" Nov 01 06:45:17.281349 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:17.281326 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5be4fa-29891925-l24ff" event={"ID":"4957cc07-721f-40ca-85e7-3d9ca330096e","Type":"ContainerStarted","Data":"3b2c0807cf7f7d838fa3fe304c8347eda51a96ae9fd03813bd73a345322c156c"} Nov 01 06:45:17.297126 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:17.297093 5040 scope.go:122] "RemoveContainer" containerID="7452b06fa7fc7fcda3af6653df1a6b37d90dac4014494280086bfaab5c530439" Nov 01 06:45:17.401763 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:17.401726 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891925-zh28x"] Nov 01 06:45:17.456905 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:17.456876 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891925-chf96" Nov 01 06:45:17.486550 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:17.486519 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891925-cx2hq" Nov 01 06:45:17.509112 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:17.509074 5040 scope.go:122] "RemoveContainer" containerID="aaebd700e48405b2ff03806b19926d13fcc79b98f1f0fca762773006721b317f" Nov 01 06:45:17.573558 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:17.573520 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c57c77c0-2554-4f46-9c1a-a8321a993953-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57c77c0-2554-4f46-9c1a-a8321a993953-maas-api-service-ca\") pod \"c57c77c0-2554-4f46-9c1a-a8321a993953\" (UID: \"c57c77c0-2554-4f46-9c1a-a8321a993953\") " Nov 01 06:45:17.573973 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:17.573573 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c4bca9d-4e46-44e4-9270-ea42ada707bb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c4bca9d-4e46-44e4-9270-ea42ada707bb-maas-api-service-ca\") pod \"1c4bca9d-4e46-44e4-9270-ea42ada707bb\" (UID: \"1c4bca9d-4e46-44e4-9270-ea42ada707bb\") " Nov 01 06:45:17.573973 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:17.573609 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c57c77c0-2554-4f46-9c1a-a8321a993953-kube-api-access-5566m\" (UniqueName: \"kubernetes.io/projected/c57c77c0-2554-4f46-9c1a-a8321a993953-kube-api-access-5566m\") pod \"c57c77c0-2554-4f46-9c1a-a8321a993953\" (UID: \"c57c77c0-2554-4f46-9c1a-a8321a993953\") " Nov 01 06:45:17.573973 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:17.573857 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c4bca9d-4e46-44e4-9270-ea42ada707bb-kube-api-access-jqgzp\" (UniqueName: \"kubernetes.io/projected/1c4bca9d-4e46-44e4-9270-ea42ada707bb-kube-api-access-jqgzp\") pod \"1c4bca9d-4e46-44e4-9270-ea42ada707bb\" (UID: \"1c4bca9d-4e46-44e4-9270-ea42ada707bb\") " Nov 01 06:45:17.575061 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:17.575032 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c57c77c0-2554-4f46-9c1a-a8321a993953-maas-api-service-ca" pod "c57c77c0-2554-4f46-9c1a-a8321a993953" (UID: "c57c77c0-2554-4f46-9c1a-a8321a993953"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:45:17.575414 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:17.575390 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c4bca9d-4e46-44e4-9270-ea42ada707bb-maas-api-service-ca" pod "1c4bca9d-4e46-44e4-9270-ea42ada707bb" (UID: "1c4bca9d-4e46-44e4-9270-ea42ada707bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:45:17.586643 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:17.582653 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c4bca9d-4e46-44e4-9270-ea42ada707bb-kube-api-access-jqgzp" pod "1c4bca9d-4e46-44e4-9270-ea42ada707bb" (UID: "1c4bca9d-4e46-44e4-9270-ea42ada707bb"). InnerVolumeSpecName "kube-api-access-jqgzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:45:17.586643 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:17.584624 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c57c77c0-2554-4f46-9c1a-a8321a993953-kube-api-access-5566m" pod "c57c77c0-2554-4f46-9c1a-a8321a993953" (UID: "c57c77c0-2554-4f46-9c1a-a8321a993953"). InnerVolumeSpecName "kube-api-access-5566m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:45:17.675802 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:17.675766 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqgzp\" (UniqueName: \"kubernetes.io/projected/1c4bca9d-4e46-44e4-9270-ea42ada707bb-kube-api-access-jqgzp\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:17.675802 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:17.675798 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57c77c0-2554-4f46-9c1a-a8321a993953-maas-api-service-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:17.675802 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:17.675811 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c4bca9d-4e46-44e4-9270-ea42ada707bb-maas-api-service-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:17.675802 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:17.675821 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5566m\" (UniqueName: \"kubernetes.io/projected/c57c77c0-2554-4f46-9c1a-a8321a993953-kube-api-access-5566m\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:17.737184 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:17.737149 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891925-5r4hb"] Nov 01 06:45:17.857742 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:17.857705 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891925-dqkfq"] Nov 01 06:45:18.295144 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:18.295093 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891925-c7wn9" event={"ID":"d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9","Type":"ContainerStarted","Data":"5ba39159759448390dda838e0b7d7ebcf6d92e260c6fc15e1ac489dc6caf8bdb"} Nov 01 06:45:18.298965 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:18.298935 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891925-chf96" Nov 01 06:45:18.299083 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:18.298943 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891925-chf96" event={"ID":"c57c77c0-2554-4f46-9c1a-a8321a993953","Type":"ContainerDied","Data":"a61dc48b12e7e4286b213acd9f38bcc3b6234e87cdd43db799d0366370bc40f4"} Nov 01 06:45:18.299141 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:18.299120 5040 scope.go:122] "RemoveContainer" containerID="0f2ce942a773ed0d1085577e333d9914a244a2b3902953fbcf3c69fd37452951" Nov 01 06:45:18.308181 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:18.308150 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891925-cx2hq" Nov 01 06:45:18.308286 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:18.308177 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891925-cx2hq" event={"ID":"1c4bca9d-4e46-44e4-9270-ea42ada707bb","Type":"ContainerDied","Data":"3a875f927888c1b88987dc61c033fdc848296fa235dedf48277b9c10b65545d2"} Nov 01 06:45:18.311237 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:18.311209 5040 generic.go:356] "Generic (PLEG): container finished" podID="e328a3ce-7f5d-4599-a616-b49f6f18a510" containerID="ca35cb5378aeda40df800668fc34c39f1f444978614dcd76d2783ec03c0e474e" exitCode=60 Nov 01 06:45:18.311360 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:18.311293 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-747b7b-29891925-g8kd9" event={"ID":"e328a3ce-7f5d-4599-a616-b49f6f18a510","Type":"ContainerDied","Data":"ca35cb5378aeda40df800668fc34c39f1f444978614dcd76d2783ec03c0e474e"} Nov 01 06:45:18.311774 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:18.311744 5040 scope.go:122] "RemoveContainer" containerID="ca35cb5378aeda40df800668fc34c39f1f444978614dcd76d2783ec03c0e474e" Nov 01 06:45:18.312049 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:45:18.312024 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-747b7b-29891925-g8kd9_odh-ai-gateway-infra(e328a3ce-7f5d-4599-a616-b49f6f18a510)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-747b7b-29891925-g8kd9" podUID="e328a3ce-7f5d-4599-a616-b49f6f18a510" Nov 01 06:45:18.313635 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:18.313614 5040 scope.go:122] "RemoveContainer" containerID="159100641f19e70b6b7d8cb9725113eeb211c2e387b4b6cfe68d34e8813781bd" Nov 01 06:45:18.315725 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:18.315694 5040 generic.go:356] "Generic (PLEG): container finished" podID="4957cc07-721f-40ca-85e7-3d9ca330096e" containerID="3b2c0807cf7f7d838fa3fe304c8347eda51a96ae9fd03813bd73a345322c156c" exitCode=60 Nov 01 06:45:18.315839 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:18.315734 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5be4fa-29891925-l24ff" event={"ID":"4957cc07-721f-40ca-85e7-3d9ca330096e","Type":"ContainerDied","Data":"3b2c0807cf7f7d838fa3fe304c8347eda51a96ae9fd03813bd73a345322c156c"} Nov 01 06:45:18.316131 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:18.316112 5040 scope.go:122] "RemoveContainer" containerID="3b2c0807cf7f7d838fa3fe304c8347eda51a96ae9fd03813bd73a345322c156c" Nov 01 06:45:18.316506 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:45:18.316446 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-5be4fa-29891925-l24ff_odh-ai-gateway-infra(4957cc07-721f-40ca-85e7-3d9ca330096e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5be4fa-29891925-l24ff" podUID="4957cc07-721f-40ca-85e7-3d9ca330096e" Nov 01 06:45:18.333715 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:18.333360 5040 scope.go:122] "RemoveContainer" containerID="5cc340120a879508fa766f7c21fea16812f4f7b2f9ab43a0ea73f268a9287746" Nov 01 06:45:18.364623 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:18.363248 5040 scope.go:122] "RemoveContainer" containerID="c5c4976b59d16061e9fc3ecf5b8f541886272d810d3b1794e8a6b2764f5c1d1e" Nov 01 06:45:18.604400 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:18.604369 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891925-zh28x" Nov 01 06:45:18.608531 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:18.608504 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891925-dqkfq" Nov 01 06:45:18.612922 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:18.612901 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891925-5r4hb" Nov 01 06:45:18.699541 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:18.699463 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0431f2e8-f3aa-4f41-9360-0f12c5baa029-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0431f2e8-f3aa-4f41-9360-0f12c5baa029-maas-api-service-ca\") pod \"0431f2e8-f3aa-4f41-9360-0f12c5baa029\" (UID: \"0431f2e8-f3aa-4f41-9360-0f12c5baa029\") " Nov 01 06:45:18.699736 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:18.699565 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7482f875-ca8d-4b85-ab43-5e7eea31efca-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7482f875-ca8d-4b85-ab43-5e7eea31efca-maas-api-service-ca\") pod \"7482f875-ca8d-4b85-ab43-5e7eea31efca\" (UID: \"7482f875-ca8d-4b85-ab43-5e7eea31efca\") " Nov 01 06:45:18.699736 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:18.699643 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0431f2e8-f3aa-4f41-9360-0f12c5baa029-kube-api-access-tpsgx\" (UniqueName: \"kubernetes.io/projected/0431f2e8-f3aa-4f41-9360-0f12c5baa029-kube-api-access-tpsgx\") pod \"0431f2e8-f3aa-4f41-9360-0f12c5baa029\" (UID: \"0431f2e8-f3aa-4f41-9360-0f12c5baa029\") " Nov 01 06:45:18.699736 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:18.699718 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a8d4067-a520-4890-9d89-e73e382438b1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a8d4067-a520-4890-9d89-e73e382438b1-maas-api-service-ca\") pod \"1a8d4067-a520-4890-9d89-e73e382438b1\" (UID: \"1a8d4067-a520-4890-9d89-e73e382438b1\") " Nov 01 06:45:18.699922 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:18.699801 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7482f875-ca8d-4b85-ab43-5e7eea31efca-kube-api-access-4cjh9\" (UniqueName: \"kubernetes.io/projected/7482f875-ca8d-4b85-ab43-5e7eea31efca-kube-api-access-4cjh9\") pod \"7482f875-ca8d-4b85-ab43-5e7eea31efca\" (UID: \"7482f875-ca8d-4b85-ab43-5e7eea31efca\") " Nov 01 06:45:18.699922 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:18.699824 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a8d4067-a520-4890-9d89-e73e382438b1-kube-api-access-rmtmp\" (UniqueName: \"kubernetes.io/projected/1a8d4067-a520-4890-9d89-e73e382438b1-kube-api-access-rmtmp\") pod \"1a8d4067-a520-4890-9d89-e73e382438b1\" (UID: \"1a8d4067-a520-4890-9d89-e73e382438b1\") " Nov 01 06:45:18.699922 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:18.699900 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0431f2e8-f3aa-4f41-9360-0f12c5baa029-maas-api-service-ca" pod "0431f2e8-f3aa-4f41-9360-0f12c5baa029" (UID: "0431f2e8-f3aa-4f41-9360-0f12c5baa029"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:45:18.700102 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:18.700018 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7482f875-ca8d-4b85-ab43-5e7eea31efca-maas-api-service-ca" pod "7482f875-ca8d-4b85-ab43-5e7eea31efca" (UID: "7482f875-ca8d-4b85-ab43-5e7eea31efca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:45:18.700159 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:18.700116 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a8d4067-a520-4890-9d89-e73e382438b1-maas-api-service-ca" pod "1a8d4067-a520-4890-9d89-e73e382438b1" (UID: "1a8d4067-a520-4890-9d89-e73e382438b1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:45:18.700356 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:18.700338 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a8d4067-a520-4890-9d89-e73e382438b1-maas-api-service-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:18.700441 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:18.700362 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0431f2e8-f3aa-4f41-9360-0f12c5baa029-maas-api-service-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:18.700441 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:18.700376 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7482f875-ca8d-4b85-ab43-5e7eea31efca-maas-api-service-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:18.702197 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:18.702169 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0431f2e8-f3aa-4f41-9360-0f12c5baa029-kube-api-access-tpsgx" pod "0431f2e8-f3aa-4f41-9360-0f12c5baa029" (UID: "0431f2e8-f3aa-4f41-9360-0f12c5baa029"). InnerVolumeSpecName "kube-api-access-tpsgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:45:18.702522 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:18.702464 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8d4067-a520-4890-9d89-e73e382438b1-kube-api-access-rmtmp" pod "1a8d4067-a520-4890-9d89-e73e382438b1" (UID: "1a8d4067-a520-4890-9d89-e73e382438b1"). InnerVolumeSpecName "kube-api-access-rmtmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:45:18.702638 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:18.702531 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7482f875-ca8d-4b85-ab43-5e7eea31efca-kube-api-access-4cjh9" pod "7482f875-ca8d-4b85-ab43-5e7eea31efca" (UID: "7482f875-ca8d-4b85-ab43-5e7eea31efca"). InnerVolumeSpecName "kube-api-access-4cjh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:45:18.801490 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:18.801438 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpsgx\" (UniqueName: \"kubernetes.io/projected/0431f2e8-f3aa-4f41-9360-0f12c5baa029-kube-api-access-tpsgx\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:18.801490 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:18.801491 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cjh9\" (UniqueName: \"kubernetes.io/projected/7482f875-ca8d-4b85-ab43-5e7eea31efca-kube-api-access-4cjh9\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:18.801711 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:18.801507 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmtmp\" (UniqueName: \"kubernetes.io/projected/1a8d4067-a520-4890-9d89-e73e382438b1-kube-api-access-rmtmp\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:18.896460 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:18.896370 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-747b7b-29891925-g8kd9"] Nov 01 06:45:19.054978 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.054938 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891925-chf96"] Nov 01 06:45:19.059265 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.059232 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d7d444-29891925-chf96"] Nov 01 06:45:19.135420 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.135373 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5be4fa-29891925-l24ff"] Nov 01 06:45:19.169908 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.169820 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891925-cx2hq"] Nov 01 06:45:19.175679 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.175640 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99e79c-29891925-cx2hq"] Nov 01 06:45:19.325980 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.325949 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891925-dqkfq" Nov 01 06:45:19.325980 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.325965 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891925-dqkfq" event={"ID":"7482f875-ca8d-4b85-ab43-5e7eea31efca","Type":"ContainerDied","Data":"2db9420c2bfd094fbd3d27de0457575c9bb581965c41c854505806c011546658"} Nov 01 06:45:19.326220 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.326032 5040 scope.go:122] "RemoveContainer" containerID="8b4c8602821104a48a2c23ef4610f5b4563f011f2ceafb37d8f0853eade6cea8" Nov 01 06:45:19.333333 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.333308 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891925-zh28x" Nov 01 06:45:19.333458 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.333327 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891925-zh28x" event={"ID":"1a8d4067-a520-4890-9d89-e73e382438b1","Type":"ContainerDied","Data":"6989d20a8a88272b64b54c772839be735ab256e60c45f065d5135f50d9e6741d"} Nov 01 06:45:19.337573 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.336201 5040 generic.go:356] "Generic (PLEG): container finished" podID="d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9" containerID="5ba39159759448390dda838e0b7d7ebcf6d92e260c6fc15e1ac489dc6caf8bdb" exitCode=60 Nov 01 06:45:19.337573 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.336268 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891925-c7wn9" event={"ID":"d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9","Type":"ContainerDied","Data":"5ba39159759448390dda838e0b7d7ebcf6d92e260c6fc15e1ac489dc6caf8bdb"} Nov 01 06:45:19.337573 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.336986 5040 scope.go:122] "RemoveContainer" containerID="5ba39159759448390dda838e0b7d7ebcf6d92e260c6fc15e1ac489dc6caf8bdb" Nov 01 06:45:19.337573 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:45:19.337237 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891925-c7wn9_odh-ai-gateway-infra(d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891925-c7wn9" podUID="d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9" Nov 01 06:45:19.343828 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.343741 5040 scope.go:122] "RemoveContainer" containerID="10928088b49718c4eb39a880a1d0b7ef97dfe8ffe6931c48ea58ac6ad456bb90" Nov 01 06:45:19.345166 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.345095 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891925-5r4hb" Nov 01 06:45:19.345166 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.345095 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891925-5r4hb" event={"ID":"0431f2e8-f3aa-4f41-9360-0f12c5baa029","Type":"ContainerDied","Data":"79248018f278fb272d67eb51bc230899c99a7cc21226665a3ed5fe6732370f5c"} Nov 01 06:45:19.361256 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.361230 5040 scope.go:122] "RemoveContainer" containerID="aaebd700e48405b2ff03806b19926d13fcc79b98f1f0fca762773006721b317f" Nov 01 06:45:19.381550 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.381504 5040 scope.go:122] "RemoveContainer" containerID="08152e853ef13fc17877f5ed433b2bd0d5602b8cd35cff3e4664289d6d064068" Nov 01 06:45:19.538429 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.536383 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c4bca9d-4e46-44e4-9270-ea42ada707bb" path="/var/lib/kubelet/pods/1c4bca9d-4e46-44e4-9270-ea42ada707bb/volumes" Nov 01 06:45:19.538429 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.537149 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c57c77c0-2554-4f46-9c1a-a8321a993953" path="/var/lib/kubelet/pods/c57c77c0-2554-4f46-9c1a-a8321a993953/volumes" Nov 01 06:45:19.552880 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.552848 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-747b7b-29891925-g8kd9" Nov 01 06:45:19.554924 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.554896 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5be4fa-29891925-l24ff" Nov 01 06:45:19.612189 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.612154 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4957cc07-721f-40ca-85e7-3d9ca330096e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4957cc07-721f-40ca-85e7-3d9ca330096e-maas-api-service-ca\") pod \"4957cc07-721f-40ca-85e7-3d9ca330096e\" (UID: \"4957cc07-721f-40ca-85e7-3d9ca330096e\") " Nov 01 06:45:19.612715 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.612228 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e328a3ce-7f5d-4599-a616-b49f6f18a510-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e328a3ce-7f5d-4599-a616-b49f6f18a510-maas-api-service-ca\") pod \"e328a3ce-7f5d-4599-a616-b49f6f18a510\" (UID: \"e328a3ce-7f5d-4599-a616-b49f6f18a510\") " Nov 01 06:45:19.612715 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.612265 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4957cc07-721f-40ca-85e7-3d9ca330096e-kube-api-access-fjf68\" (UniqueName: \"kubernetes.io/projected/4957cc07-721f-40ca-85e7-3d9ca330096e-kube-api-access-fjf68\") pod \"4957cc07-721f-40ca-85e7-3d9ca330096e\" (UID: \"4957cc07-721f-40ca-85e7-3d9ca330096e\") " Nov 01 06:45:19.612715 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.612284 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e328a3ce-7f5d-4599-a616-b49f6f18a510-kube-api-access-fvnd8\" (UniqueName: \"kubernetes.io/projected/e328a3ce-7f5d-4599-a616-b49f6f18a510-kube-api-access-fvnd8\") pod \"e328a3ce-7f5d-4599-a616-b49f6f18a510\" (UID: \"e328a3ce-7f5d-4599-a616-b49f6f18a510\") " Nov 01 06:45:19.612715 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.612627 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4957cc07-721f-40ca-85e7-3d9ca330096e-maas-api-service-ca" pod "4957cc07-721f-40ca-85e7-3d9ca330096e" (UID: "4957cc07-721f-40ca-85e7-3d9ca330096e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:45:19.612715 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.612628 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e328a3ce-7f5d-4599-a616-b49f6f18a510-maas-api-service-ca" pod "e328a3ce-7f5d-4599-a616-b49f6f18a510" (UID: "e328a3ce-7f5d-4599-a616-b49f6f18a510"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:45:19.613060 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.613037 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e328a3ce-7f5d-4599-a616-b49f6f18a510-maas-api-service-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:19.613171 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.613135 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4957cc07-721f-40ca-85e7-3d9ca330096e-maas-api-service-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:19.614795 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.614766 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e328a3ce-7f5d-4599-a616-b49f6f18a510-kube-api-access-fvnd8" pod "e328a3ce-7f5d-4599-a616-b49f6f18a510" (UID: "e328a3ce-7f5d-4599-a616-b49f6f18a510"). InnerVolumeSpecName "kube-api-access-fvnd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:45:19.614923 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.614903 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4957cc07-721f-40ca-85e7-3d9ca330096e-kube-api-access-fjf68" pod "4957cc07-721f-40ca-85e7-3d9ca330096e" (UID: "4957cc07-721f-40ca-85e7-3d9ca330096e"). InnerVolumeSpecName "kube-api-access-fjf68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:45:19.713908 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.713816 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjf68\" (UniqueName: \"kubernetes.io/projected/4957cc07-721f-40ca-85e7-3d9ca330096e-kube-api-access-fjf68\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:19.713908 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.713849 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvnd8\" (UniqueName: \"kubernetes.io/projected/e328a3ce-7f5d-4599-a616-b49f6f18a510-kube-api-access-fvnd8\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:19.760662 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.760617 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891925-5r4hb"] Nov 01 06:45:19.766845 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.766811 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891925-5r4hb"] Nov 01 06:45:19.951270 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.951235 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891925-c7wn9"] Nov 01 06:45:19.996152 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:19.996077 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891925-zh28x"] Nov 01 06:45:20.001881 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:20.001852 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-71cea4-29891925-zh28x"] Nov 01 06:45:20.110344 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:20.110293 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891925-dqkfq"] Nov 01 06:45:20.115654 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:20.115623 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3986bf-29891925-dqkfq"] Nov 01 06:45:20.360376 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:20.360343 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-747b7b-29891925-g8kd9" Nov 01 06:45:20.360607 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:20.360344 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-747b7b-29891925-g8kd9" event={"ID":"e328a3ce-7f5d-4599-a616-b49f6f18a510","Type":"ContainerDied","Data":"5cd313306d2ba9d2b8cd81c97395e395012f4ad66222db5cd22f53b4e927b28a"} Nov 01 06:45:20.360607 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:20.360506 5040 scope.go:122] "RemoveContainer" containerID="ca35cb5378aeda40df800668fc34c39f1f444978614dcd76d2783ec03c0e474e" Nov 01 06:45:20.362242 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:20.362221 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5be4fa-29891925-l24ff" Nov 01 06:45:20.362359 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:20.362243 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5be4fa-29891925-l24ff" event={"ID":"4957cc07-721f-40ca-85e7-3d9ca330096e","Type":"ContainerDied","Data":"f9bcb225d331f300f6a0b21ba748b7d46a3764de91e8195dd2b0f08cf172e329"} Nov 01 06:45:20.375414 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:20.375395 5040 scope.go:122] "RemoveContainer" containerID="3b2c0807cf7f7d838fa3fe304c8347eda51a96ae9fd03813bd73a345322c156c" Nov 01 06:45:20.533809 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:20.533780 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891925-c7wn9" Nov 01 06:45:20.536305 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:20.536282 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5be4fa-29891925-l24ff"] Nov 01 06:45:20.546575 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:20.546545 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5be4fa-29891925-l24ff"] Nov 01 06:45:20.624307 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:20.624228 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9-maas-api-service-ca\") pod \"d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9\" (UID: \"d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9\") " Nov 01 06:45:20.624736 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:20.624325 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9-kube-api-access-gb722\" (UniqueName: \"kubernetes.io/projected/d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9-kube-api-access-gb722\") pod \"d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9\" (UID: \"d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9\") " Nov 01 06:45:20.624808 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:20.624734 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9-maas-api-service-ca" pod "d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9" (UID: "d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:45:20.625026 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:20.625004 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9-maas-api-service-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:20.626573 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:20.626546 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9-kube-api-access-gb722" pod "d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9" (UID: "d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9"). InnerVolumeSpecName "kube-api-access-gb722". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:45:20.726179 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:20.726143 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb722\" (UniqueName: \"kubernetes.io/projected/d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9-kube-api-access-gb722\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:20.769743 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:20.769701 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-747b7b-29891925-g8kd9"] Nov 01 06:45:20.772861 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:20.772838 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-747b7b-29891925-g8kd9"] Nov 01 06:45:21.049488 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:21.049451 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3596b654-cm8gq"] Nov 01 06:45:21.052647 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:21.052624 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3596b654-cm8gq"] Nov 01 06:45:21.377466 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:21.377444 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891925-c7wn9" Nov 01 06:45:21.377653 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:21.377474 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891925-c7wn9" event={"ID":"d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9","Type":"ContainerDied","Data":"dfbda5c38d4d60ecf127ea158709e0eb591bf7ac2c2aa4ccb1233bb40349c838"} Nov 01 06:45:21.377653 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:21.377550 5040 scope.go:122] "RemoveContainer" containerID="5ba39159759448390dda838e0b7d7ebcf6d92e260c6fc15e1ac489dc6caf8bdb" Nov 01 06:45:21.379262 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:21.379241 5040 generic.go:356] "Generic (PLEG): container finished" podID="cee73e53-f9e2-49c9-a19c-ee33cc2ac68e" containerID="1a88b3ba3539e775c2940edeeb343c0a783531829cc0618570eb68cbb26bfd64" exitCode=0 Nov 01 06:45:21.379363 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:21.379312 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" event={"ID":"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e","Type":"ContainerDied","Data":"1a88b3ba3539e775c2940edeeb343c0a783531829cc0618570eb68cbb26bfd64"} Nov 01 06:45:21.379732 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:21.379711 5040 scope.go:122] "RemoveContainer" containerID="1a88b3ba3539e775c2940edeeb343c0a783531829cc0618570eb68cbb26bfd64" Nov 01 06:45:21.381347 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:21.381317 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:45:21.381457 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:21.381400 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:45:21.381457 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:21.381450 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:45:21.516969 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:21.516890 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0431f2e8-f3aa-4f41-9360-0f12c5baa029" path="/var/lib/kubelet/pods/0431f2e8-f3aa-4f41-9360-0f12c5baa029/volumes" Nov 01 06:45:21.517464 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:21.517435 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a8d4067-a520-4890-9d89-e73e382438b1" path="/var/lib/kubelet/pods/1a8d4067-a520-4890-9d89-e73e382438b1/volumes" Nov 01 06:45:21.517977 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:21.517958 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4957cc07-721f-40ca-85e7-3d9ca330096e" path="/var/lib/kubelet/pods/4957cc07-721f-40ca-85e7-3d9ca330096e/volumes" Nov 01 06:45:21.518804 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:21.518782 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7482f875-ca8d-4b85-ab43-5e7eea31efca" path="/var/lib/kubelet/pods/7482f875-ca8d-4b85-ab43-5e7eea31efca/volumes" Nov 01 06:45:21.519185 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:21.519170 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e328a3ce-7f5d-4599-a616-b49f6f18a510" path="/var/lib/kubelet/pods/e328a3ce-7f5d-4599-a616-b49f6f18a510/volumes" Nov 01 06:45:21.519595 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:21.519580 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6cd3d93-fb67-468d-a3af-98a4a460fe82" path="/var/lib/kubelet/pods/e6cd3d93-fb67-468d-a3af-98a4a460fe82/volumes" Nov 01 06:45:21.532699 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:21.532675 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:45:21.670538 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:21.670406 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891925-c7wn9"] Nov 01 06:45:21.674498 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:21.674455 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891925-c7wn9"] Nov 01 06:45:22.202355 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:22.202327 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 06:45:22.205955 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:22.205926 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 06:45:22.391715 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:22.391677 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" event={"ID":"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e","Type":"ContainerStarted","Data":"560e4e560172830fcd6e0f82fda9e50047d90419514418280660f6941eab990d"} Nov 01 06:45:22.521594 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:22.521363 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:45:22.522439 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:22.522395 5040 patch_prober.go:29] interesting pod/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Nov 01 06:45:22.522707 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:22.522455 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" podUID="cee73e53-f9e2-49c9-a19c-ee33cc2ac68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Nov 01 06:45:23.520505 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:23.518813 5040 patch_prober.go:29] interesting pod/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Nov 01 06:45:23.520505 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:23.518875 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" podUID="cee73e53-f9e2-49c9-a19c-ee33cc2ac68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Nov 01 06:45:23.528187 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:23.528154 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9" path="/var/lib/kubelet/pods/d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9/volumes" Nov 01 06:45:23.628454 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:23.628423 5040 scope.go:122] "RemoveContainer" containerID="1f00849015fa332305dc38a67009a848fa048abe9090414d368d90b884c26e60" Nov 01 06:45:23.638431 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:23.638410 5040 scope.go:122] "RemoveContainer" containerID="0d1f0560b5595f08203d97477b116efe5761c5b227e85f07d7907e7b04d6b253" Nov 01 06:45:24.520502 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:24.517832 5040 patch_prober.go:29] interesting pod/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Nov 01 06:45:24.520502 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:24.517907 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" podUID="cee73e53-f9e2-49c9-a19c-ee33cc2ac68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Nov 01 06:45:25.519607 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:25.519566 5040 patch_prober.go:29] interesting pod/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Nov 01 06:45:25.519934 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:25.519900 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" podUID="cee73e53-f9e2-49c9-a19c-ee33cc2ac68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Nov 01 06:45:26.048807 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:26.048764 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5b2be7c-vnmkb"] Nov 01 06:45:26.120199 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:26.120152 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5b2be7c-vnmkb"] Nov 01 06:45:26.518882 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:26.518836 5040 patch_prober.go:29] interesting pod/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Nov 01 06:45:26.519105 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:26.518902 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" podUID="cee73e53-f9e2-49c9-a19c-ee33cc2ac68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Nov 01 06:45:27.519203 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:27.519053 5040 patch_prober.go:29] interesting pod/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Nov 01 06:45:27.519203 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:27.519112 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" podUID="cee73e53-f9e2-49c9-a19c-ee33cc2ac68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Nov 01 06:45:27.524230 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:27.524196 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c71b282f-ff6a-4ba8-aec0-2f651770e690" path="/var/lib/kubelet/pods/c71b282f-ff6a-4ba8-aec0-2f651770e690/volumes" Nov 01 06:45:28.521925 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:28.521893 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:45:28.522352 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:28.522314 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:45:28.523007 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:28.522990 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:45:39.974281 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:45:39.974242 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:45:47.888137 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.888094 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a1f9505-v49hm"] Nov 01 06:45:47.888855 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.888831 5040 cpu_manager.go:414] "containerMap: removing container" podUID="1a8d4067-a520-4890-9d89-e73e382438b1" containerName="cleanup" Nov 01 06:45:47.888957 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.888857 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a8d4067-a520-4890-9d89-e73e382438b1" containerName="cleanup" Nov 01 06:45:47.888957 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.888876 5040 cpu_manager.go:414] "containerMap: removing container" podUID="1c4bca9d-4e46-44e4-9270-ea42ada707bb" containerName="cleanup" Nov 01 06:45:47.888957 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.888883 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c4bca9d-4e46-44e4-9270-ea42ada707bb" containerName="cleanup" Nov 01 06:45:47.888957 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.888895 5040 cpu_manager.go:414] "containerMap: removing container" podUID="0431f2e8-f3aa-4f41-9360-0f12c5baa029" containerName="cleanup" Nov 01 06:45:47.888957 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.888904 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0431f2e8-f3aa-4f41-9360-0f12c5baa029" containerName="cleanup" Nov 01 06:45:47.888957 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.888919 5040 cpu_manager.go:414] "containerMap: removing container" podUID="c57c77c0-2554-4f46-9c1a-a8321a993953" containerName="cleanup" Nov 01 06:45:47.888957 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.888926 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c57c77c0-2554-4f46-9c1a-a8321a993953" containerName="cleanup" Nov 01 06:45:47.888957 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.888939 5040 cpu_manager.go:414] "containerMap: removing container" podUID="1a8d4067-a520-4890-9d89-e73e382438b1" containerName="cleanup" Nov 01 06:45:47.888957 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.888948 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a8d4067-a520-4890-9d89-e73e382438b1" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.888963 5040 cpu_manager.go:414] "containerMap: removing container" podUID="d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.888972 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.888988 5040 cpu_manager.go:414] "containerMap: removing container" podUID="0431f2e8-f3aa-4f41-9360-0f12c5baa029" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.888995 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0431f2e8-f3aa-4f41-9360-0f12c5baa029" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889008 5040 cpu_manager.go:414] "containerMap: removing container" podUID="c57c77c0-2554-4f46-9c1a-a8321a993953" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889015 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c57c77c0-2554-4f46-9c1a-a8321a993953" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889027 5040 cpu_manager.go:414] "containerMap: removing container" podUID="d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889035 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889047 5040 cpu_manager.go:414] "containerMap: removing container" podUID="4957cc07-721f-40ca-85e7-3d9ca330096e" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889054 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4957cc07-721f-40ca-85e7-3d9ca330096e" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889065 5040 cpu_manager.go:414] "containerMap: removing container" podUID="e328a3ce-7f5d-4599-a616-b49f6f18a510" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889072 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e328a3ce-7f5d-4599-a616-b49f6f18a510" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889083 5040 cpu_manager.go:414] "containerMap: removing container" podUID="4957cc07-721f-40ca-85e7-3d9ca330096e" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889091 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4957cc07-721f-40ca-85e7-3d9ca330096e" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889105 5040 cpu_manager.go:414] "containerMap: removing container" podUID="c57c77c0-2554-4f46-9c1a-a8321a993953" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889112 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c57c77c0-2554-4f46-9c1a-a8321a993953" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889122 5040 cpu_manager.go:414] "containerMap: removing container" podUID="e328a3ce-7f5d-4599-a616-b49f6f18a510" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889130 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e328a3ce-7f5d-4599-a616-b49f6f18a510" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889144 5040 cpu_manager.go:414] "containerMap: removing container" podUID="1c4bca9d-4e46-44e4-9270-ea42ada707bb" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889152 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c4bca9d-4e46-44e4-9270-ea42ada707bb" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889165 5040 cpu_manager.go:414] "containerMap: removing container" podUID="7482f875-ca8d-4b85-ab43-5e7eea31efca" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889172 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7482f875-ca8d-4b85-ab43-5e7eea31efca" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889183 5040 cpu_manager.go:414] "containerMap: removing container" podUID="1c4bca9d-4e46-44e4-9270-ea42ada707bb" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889191 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c4bca9d-4e46-44e4-9270-ea42ada707bb" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889203 5040 cpu_manager.go:414] "containerMap: removing container" podUID="e328a3ce-7f5d-4599-a616-b49f6f18a510" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889210 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e328a3ce-7f5d-4599-a616-b49f6f18a510" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889223 5040 cpu_manager.go:414] "containerMap: removing container" podUID="7482f875-ca8d-4b85-ab43-5e7eea31efca" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889230 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7482f875-ca8d-4b85-ab43-5e7eea31efca" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889260 5040 cpu_manager.go:414] "containerMap: removing container" podUID="7482f875-ca8d-4b85-ab43-5e7eea31efca" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889267 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7482f875-ca8d-4b85-ab43-5e7eea31efca" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889390 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="c57c77c0-2554-4f46-9c1a-a8321a993953" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889405 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="e328a3ce-7f5d-4599-a616-b49f6f18a510" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889416 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c4bca9d-4e46-44e4-9270-ea42ada707bb" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889425 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="7482f875-ca8d-4b85-ab43-5e7eea31efca" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889433 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="e328a3ce-7f5d-4599-a616-b49f6f18a510" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889442 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="4957cc07-721f-40ca-85e7-3d9ca330096e" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889450 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9" containerName="cleanup" Nov 01 06:45:47.889436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889457 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="7482f875-ca8d-4b85-ab43-5e7eea31efca" containerName="cleanup" Nov 01 06:45:47.891357 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889468 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a8d4067-a520-4890-9d89-e73e382438b1" containerName="cleanup" Nov 01 06:45:47.891357 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889493 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="0431f2e8-f3aa-4f41-9360-0f12c5baa029" containerName="cleanup" Nov 01 06:45:47.891357 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889503 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a8d4067-a520-4890-9d89-e73e382438b1" containerName="cleanup" Nov 01 06:45:47.891357 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889513 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c4bca9d-4e46-44e4-9270-ea42ada707bb" containerName="cleanup" Nov 01 06:45:47.891357 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889522 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="0431f2e8-f3aa-4f41-9360-0f12c5baa029" containerName="cleanup" Nov 01 06:45:47.891357 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889533 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a8d4067-a520-4890-9d89-e73e382438b1" containerName="cleanup" Nov 01 06:45:47.891357 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889543 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9" containerName="cleanup" Nov 01 06:45:47.891357 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889554 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="c57c77c0-2554-4f46-9c1a-a8321a993953" containerName="cleanup" Nov 01 06:45:47.891357 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889561 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9" containerName="cleanup" Nov 01 06:45:47.891357 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.889571 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="4957cc07-721f-40ca-85e7-3d9ca330096e" containerName="cleanup" Nov 01 06:45:47.894448 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.894425 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a1f9505-v49hm" Nov 01 06:45:47.949859 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.949824 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a1f9505-v49hm"] Nov 01 06:45:47.958906 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.958879 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpp4w\" (UniqueName: \"kubernetes.io/projected/4324605c-1796-42fa-adeb-b6f401f43645-kube-api-access-dpp4w\") pod \"maas-api-revoke-sub-5a1f9505-v49hm\" (UID: \"4324605c-1796-42fa-adeb-b6f401f43645\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a1f9505-v49hm" Nov 01 06:45:47.959136 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.958936 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4324605c-1796-42fa-adeb-b6f401f43645-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a1f9505-v49hm\" (UID: \"4324605c-1796-42fa-adeb-b6f401f43645\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a1f9505-v49hm" Nov 01 06:45:47.959136 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.959118 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4324605c-1796-42fa-adeb-b6f401f43645-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a1f9505-v49hm\" (UID: \"4324605c-1796-42fa-adeb-b6f401f43645\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a1f9505-v49hm" Nov 01 06:45:47.995389 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.995350 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:45:47.995574 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:47.995407 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:45:48.019772 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:48.019741 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:45:48.019937 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:48.019869 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:45:48.060676 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:48.060644 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpp4w\" (UniqueName: \"kubernetes.io/projected/4324605c-1796-42fa-adeb-b6f401f43645-kube-api-access-dpp4w\") pod \"maas-api-revoke-sub-5a1f9505-v49hm\" (UID: \"4324605c-1796-42fa-adeb-b6f401f43645\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a1f9505-v49hm" Nov 01 06:45:48.060874 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:48.060704 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4324605c-1796-42fa-adeb-b6f401f43645-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a1f9505-v49hm\" (UID: \"4324605c-1796-42fa-adeb-b6f401f43645\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a1f9505-v49hm" Nov 01 06:45:48.060874 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:48.060786 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4324605c-1796-42fa-adeb-b6f401f43645-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a1f9505-v49hm\" (UID: \"4324605c-1796-42fa-adeb-b6f401f43645\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a1f9505-v49hm" Nov 01 06:45:48.061595 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:48.061575 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4324605c-1796-42fa-adeb-b6f401f43645-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a1f9505-v49hm\" (UID: \"4324605c-1796-42fa-adeb-b6f401f43645\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a1f9505-v49hm" Nov 01 06:45:48.063154 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:48.063135 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4324605c-1796-42fa-adeb-b6f401f43645-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a1f9505-v49hm\" (UID: \"4324605c-1796-42fa-adeb-b6f401f43645\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a1f9505-v49hm" Nov 01 06:45:48.122316 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:48.122280 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpp4w\" (UniqueName: \"kubernetes.io/projected/4324605c-1796-42fa-adeb-b6f401f43645-kube-api-access-dpp4w\") pod \"maas-api-revoke-sub-5a1f9505-v49hm\" (UID: \"4324605c-1796-42fa-adeb-b6f401f43645\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a1f9505-v49hm" Nov 01 06:45:48.207300 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:48.207212 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a1f9505-v49hm" Nov 01 06:45:48.589855 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:48.589818 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a1f9505-v49hm"] Nov 01 06:45:48.607302 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:45:48.607273 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4324605c_1796_42fa_adeb_b6f401f43645.slice/crio-674c73f57d1d225cbb3630f11e6c5eccd1da06bc1ae6bda29fc3b12e615e0ff9 WatchSource:0}: Error finding container 674c73f57d1d225cbb3630f11e6c5eccd1da06bc1ae6bda29fc3b12e615e0ff9: Status 404 returned error can't find the container with id 674c73f57d1d225cbb3630f11e6c5eccd1da06bc1ae6bda29fc3b12e615e0ff9 Nov 01 06:45:49.609898 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:49.609863 5040 generic.go:356] "Generic (PLEG): container finished" podID="4324605c-1796-42fa-adeb-b6f401f43645" containerID="67802d218ad04ddf14dbb05ba0440793aff3438620c89fa036f6205369735508" exitCode=0 Nov 01 06:45:49.610290 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:49.609944 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a1f9505-v49hm" event={"ID":"4324605c-1796-42fa-adeb-b6f401f43645","Type":"ContainerDied","Data":"67802d218ad04ddf14dbb05ba0440793aff3438620c89fa036f6205369735508"} Nov 01 06:45:49.610290 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:49.609997 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a1f9505-v49hm" event={"ID":"4324605c-1796-42fa-adeb-b6f401f43645","Type":"ContainerStarted","Data":"674c73f57d1d225cbb3630f11e6c5eccd1da06bc1ae6bda29fc3b12e615e0ff9"} Nov 01 06:45:50.766115 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:50.766087 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a1f9505-v49hm" Nov 01 06:45:50.891898 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:50.891809 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4324605c-1796-42fa-adeb-b6f401f43645-kube-api-access-dpp4w\" (UniqueName: \"kubernetes.io/projected/4324605c-1796-42fa-adeb-b6f401f43645-kube-api-access-dpp4w\") pod \"4324605c-1796-42fa-adeb-b6f401f43645\" (UID: \"4324605c-1796-42fa-adeb-b6f401f43645\") " Nov 01 06:45:50.891898 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:50.891887 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4324605c-1796-42fa-adeb-b6f401f43645-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4324605c-1796-42fa-adeb-b6f401f43645-maas-api-xks-ca\") pod \"4324605c-1796-42fa-adeb-b6f401f43645\" (UID: \"4324605c-1796-42fa-adeb-b6f401f43645\") " Nov 01 06:45:50.892105 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:50.892063 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4324605c-1796-42fa-adeb-b6f401f43645-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4324605c-1796-42fa-adeb-b6f401f43645-maas-api-openshift-ca\") pod \"4324605c-1796-42fa-adeb-b6f401f43645\" (UID: \"4324605c-1796-42fa-adeb-b6f401f43645\") " Nov 01 06:45:50.892413 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:50.892391 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4324605c-1796-42fa-adeb-b6f401f43645-maas-api-openshift-ca" pod "4324605c-1796-42fa-adeb-b6f401f43645" (UID: "4324605c-1796-42fa-adeb-b6f401f43645"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:45:50.892568 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:50.892545 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4324605c-1796-42fa-adeb-b6f401f43645-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:50.894270 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:50.894241 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4324605c-1796-42fa-adeb-b6f401f43645-kube-api-access-dpp4w" pod "4324605c-1796-42fa-adeb-b6f401f43645" (UID: "4324605c-1796-42fa-adeb-b6f401f43645"). InnerVolumeSpecName "kube-api-access-dpp4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:45:50.894386 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:50.894244 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4324605c-1796-42fa-adeb-b6f401f43645-maas-api-xks-ca" pod "4324605c-1796-42fa-adeb-b6f401f43645" (UID: "4324605c-1796-42fa-adeb-b6f401f43645"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:45:50.993632 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:50.993577 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpp4w\" (UniqueName: \"kubernetes.io/projected/4324605c-1796-42fa-adeb-b6f401f43645-kube-api-access-dpp4w\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:50.993632 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:50.993627 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4324605c-1796-42fa-adeb-b6f401f43645-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:51.624748 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:51.624710 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a1f9505-v49hm" event={"ID":"4324605c-1796-42fa-adeb-b6f401f43645","Type":"ContainerDied","Data":"674c73f57d1d225cbb3630f11e6c5eccd1da06bc1ae6bda29fc3b12e615e0ff9"} Nov 01 06:45:51.624920 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:51.624757 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a1f9505-v49hm" Nov 01 06:45:51.624920 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:45:51.624772 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="674c73f57d1d225cbb3630f11e6c5eccd1da06bc1ae6bda29fc3b12e615e0ff9" Nov 01 06:46:04.145824 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:04.145785 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09e9f1d9-wwzjb"] Nov 01 06:46:04.147750 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:04.147723 5040 cpu_manager.go:414] "containerMap: removing container" podUID="4324605c-1796-42fa-adeb-b6f401f43645" containerName="revoke-keys" Nov 01 06:46:04.148337 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:04.148316 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4324605c-1796-42fa-adeb-b6f401f43645" containerName="revoke-keys" Nov 01 06:46:04.148460 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:04.148448 5040 cpu_manager.go:414] "containerMap: removing container" podUID="1a8d4067-a520-4890-9d89-e73e382438b1" containerName="cleanup" Nov 01 06:46:04.148566 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:04.148553 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a8d4067-a520-4890-9d89-e73e382438b1" containerName="cleanup" Nov 01 06:46:04.148667 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:04.148656 5040 cpu_manager.go:414] "containerMap: removing container" podUID="0431f2e8-f3aa-4f41-9360-0f12c5baa029" containerName="cleanup" Nov 01 06:46:04.148745 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:04.148735 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0431f2e8-f3aa-4f41-9360-0f12c5baa029" containerName="cleanup" Nov 01 06:46:04.148832 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:04.148823 5040 cpu_manager.go:414] "containerMap: removing container" podUID="d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9" containerName="cleanup" Nov 01 06:46:04.148907 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:04.148897 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a4fd5f-bf37-4bef-afa0-46ecafe6b8a9" containerName="cleanup" Nov 01 06:46:04.148996 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:04.148987 5040 cpu_manager.go:414] "containerMap: removing container" podUID="4957cc07-721f-40ca-85e7-3d9ca330096e" containerName="cleanup" Nov 01 06:46:04.149066 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:04.149057 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4957cc07-721f-40ca-85e7-3d9ca330096e" containerName="cleanup" Nov 01 06:46:04.149393 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:04.149377 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c4bca9d-4e46-44e4-9270-ea42ada707bb" containerName="cleanup" Nov 01 06:46:04.149536 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:04.149523 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="0431f2e8-f3aa-4f41-9360-0f12c5baa029" containerName="cleanup" Nov 01 06:46:04.149637 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:04.149627 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="4324605c-1796-42fa-adeb-b6f401f43645" containerName="revoke-keys" Nov 01 06:46:04.149726 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:04.149717 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="c57c77c0-2554-4f46-9c1a-a8321a993953" containerName="cleanup" Nov 01 06:46:04.149810 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:04.149800 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="e328a3ce-7f5d-4599-a616-b49f6f18a510" containerName="cleanup" Nov 01 06:46:04.149886 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:04.149877 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="4957cc07-721f-40ca-85e7-3d9ca330096e" containerName="cleanup" Nov 01 06:46:04.149968 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:04.149959 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="7482f875-ca8d-4b85-ab43-5e7eea31efca" containerName="cleanup" Nov 01 06:46:04.154719 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:04.154694 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e9f1d9-wwzjb" Nov 01 06:46:04.204829 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:04.204795 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09e9f1d9-wwzjb"] Nov 01 06:46:04.207728 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:04.207695 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:46:04.251159 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:04.251127 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:46:04.251159 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:04.251127 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:46:04.251428 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:04.251162 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:46:04.254739 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:04.254716 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5g6l\" (UniqueName: \"kubernetes.io/projected/2d2a1aa7-fd02-49dd-957b-8659f4299990-kube-api-access-s5g6l\") pod \"maas-api-revoke-sub-09e9f1d9-wwzjb\" (UID: \"2d2a1aa7-fd02-49dd-957b-8659f4299990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e9f1d9-wwzjb" Nov 01 06:46:04.254840 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:04.254789 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2a1aa7-fd02-49dd-957b-8659f4299990-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09e9f1d9-wwzjb\" (UID: \"2d2a1aa7-fd02-49dd-957b-8659f4299990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e9f1d9-wwzjb" Nov 01 06:46:04.254947 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:04.254929 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2a1aa7-fd02-49dd-957b-8659f4299990-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09e9f1d9-wwzjb\" (UID: \"2d2a1aa7-fd02-49dd-957b-8659f4299990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e9f1d9-wwzjb" Nov 01 06:46:04.356606 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:04.356570 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5g6l\" (UniqueName: \"kubernetes.io/projected/2d2a1aa7-fd02-49dd-957b-8659f4299990-kube-api-access-s5g6l\") pod \"maas-api-revoke-sub-09e9f1d9-wwzjb\" (UID: \"2d2a1aa7-fd02-49dd-957b-8659f4299990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e9f1d9-wwzjb" Nov 01 06:46:04.356796 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:04.356646 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2a1aa7-fd02-49dd-957b-8659f4299990-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09e9f1d9-wwzjb\" (UID: \"2d2a1aa7-fd02-49dd-957b-8659f4299990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e9f1d9-wwzjb" Nov 01 06:46:04.356796 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:04.356716 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2a1aa7-fd02-49dd-957b-8659f4299990-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09e9f1d9-wwzjb\" (UID: \"2d2a1aa7-fd02-49dd-957b-8659f4299990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e9f1d9-wwzjb" Nov 01 06:46:04.357383 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:04.357359 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2a1aa7-fd02-49dd-957b-8659f4299990-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09e9f1d9-wwzjb\" (UID: \"2d2a1aa7-fd02-49dd-957b-8659f4299990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e9f1d9-wwzjb" Nov 01 06:46:04.359093 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:04.359070 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2a1aa7-fd02-49dd-957b-8659f4299990-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09e9f1d9-wwzjb\" (UID: \"2d2a1aa7-fd02-49dd-957b-8659f4299990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e9f1d9-wwzjb" Nov 01 06:46:04.418774 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:04.418684 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5g6l\" (UniqueName: \"kubernetes.io/projected/2d2a1aa7-fd02-49dd-957b-8659f4299990-kube-api-access-s5g6l\") pod \"maas-api-revoke-sub-09e9f1d9-wwzjb\" (UID: \"2d2a1aa7-fd02-49dd-957b-8659f4299990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e9f1d9-wwzjb" Nov 01 06:46:04.468447 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:04.468408 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e9f1d9-wwzjb" Nov 01 06:46:04.851322 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:04.851284 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09e9f1d9-wwzjb"] Nov 01 06:46:04.851448 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:46:04.851341 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d2a1aa7_fd02_49dd_957b_8659f4299990.slice/crio-d9e64c99b00ddc0b913e12ab6b6217a068f132462512ce437fc97883b3461f29 WatchSource:0}: Error finding container d9e64c99b00ddc0b913e12ab6b6217a068f132462512ce437fc97883b3461f29: Status 404 returned error can't find the container with id d9e64c99b00ddc0b913e12ab6b6217a068f132462512ce437fc97883b3461f29 Nov 01 06:46:05.725211 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:05.725173 5040 generic.go:356] "Generic (PLEG): container finished" podID="2d2a1aa7-fd02-49dd-957b-8659f4299990" containerID="dfd0a051d16610fe0e76b6834add93c99ddde51a0cd75c75eccb31d1238d7d82" exitCode=0 Nov 01 06:46:05.725625 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:05.725259 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e9f1d9-wwzjb" event={"ID":"2d2a1aa7-fd02-49dd-957b-8659f4299990","Type":"ContainerDied","Data":"dfd0a051d16610fe0e76b6834add93c99ddde51a0cd75c75eccb31d1238d7d82"} Nov 01 06:46:05.725625 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:05.725317 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e9f1d9-wwzjb" event={"ID":"2d2a1aa7-fd02-49dd-957b-8659f4299990","Type":"ContainerStarted","Data":"d9e64c99b00ddc0b913e12ab6b6217a068f132462512ce437fc97883b3461f29"} Nov 01 06:46:06.965320 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:06.965293 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e9f1d9-wwzjb" Nov 01 06:46:06.998990 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:06.998957 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d2a1aa7-fd02-49dd-957b-8659f4299990-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2a1aa7-fd02-49dd-957b-8659f4299990-maas-api-xks-ca\") pod \"2d2a1aa7-fd02-49dd-957b-8659f4299990\" (UID: \"2d2a1aa7-fd02-49dd-957b-8659f4299990\") " Nov 01 06:46:06.999205 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:06.999130 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d2a1aa7-fd02-49dd-957b-8659f4299990-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2a1aa7-fd02-49dd-957b-8659f4299990-maas-api-openshift-ca\") pod \"2d2a1aa7-fd02-49dd-957b-8659f4299990\" (UID: \"2d2a1aa7-fd02-49dd-957b-8659f4299990\") " Nov 01 06:46:06.999295 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:06.999232 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d2a1aa7-fd02-49dd-957b-8659f4299990-kube-api-access-s5g6l\" (UniqueName: \"kubernetes.io/projected/2d2a1aa7-fd02-49dd-957b-8659f4299990-kube-api-access-s5g6l\") pod \"2d2a1aa7-fd02-49dd-957b-8659f4299990\" (UID: \"2d2a1aa7-fd02-49dd-957b-8659f4299990\") " Nov 01 06:46:06.999556 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:06.999508 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d2a1aa7-fd02-49dd-957b-8659f4299990-maas-api-openshift-ca" pod "2d2a1aa7-fd02-49dd-957b-8659f4299990" (UID: "2d2a1aa7-fd02-49dd-957b-8659f4299990"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:46:06.999654 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:06.999604 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2a1aa7-fd02-49dd-957b-8659f4299990-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:46:07.001150 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:07.001124 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d2a1aa7-fd02-49dd-957b-8659f4299990-maas-api-xks-ca" pod "2d2a1aa7-fd02-49dd-957b-8659f4299990" (UID: "2d2a1aa7-fd02-49dd-957b-8659f4299990"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:46:07.001284 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:07.001260 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2a1aa7-fd02-49dd-957b-8659f4299990-kube-api-access-s5g6l" pod "2d2a1aa7-fd02-49dd-957b-8659f4299990" (UID: "2d2a1aa7-fd02-49dd-957b-8659f4299990"). InnerVolumeSpecName "kube-api-access-s5g6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:46:07.100883 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:07.100839 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5g6l\" (UniqueName: \"kubernetes.io/projected/2d2a1aa7-fd02-49dd-957b-8659f4299990-kube-api-access-s5g6l\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:46:07.100883 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:07.100871 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2a1aa7-fd02-49dd-957b-8659f4299990-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:46:07.745398 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:07.745365 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e9f1d9-wwzjb" Nov 01 06:46:07.745626 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:07.745464 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e9f1d9-wwzjb" event={"ID":"2d2a1aa7-fd02-49dd-957b-8659f4299990","Type":"ContainerDied","Data":"d9e64c99b00ddc0b913e12ab6b6217a068f132462512ce437fc97883b3461f29"} Nov 01 06:46:07.745626 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:07.745537 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9e64c99b00ddc0b913e12ab6b6217a068f132462512ce437fc97883b3461f29" Nov 01 06:46:09.974667 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:46:09.974619 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:46:23.027416 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.027381 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv"] Nov 01 06:46:23.027913 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.027898 5040 cpu_manager.go:414] "containerMap: removing container" podUID="2d2a1aa7-fd02-49dd-957b-8659f4299990" containerName="revoke-keys" Nov 01 06:46:23.027961 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.027915 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d2a1aa7-fd02-49dd-957b-8659f4299990" containerName="revoke-keys" Nov 01 06:46:23.028011 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.028002 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d2a1aa7-fd02-49dd-957b-8659f4299990" containerName="revoke-keys" Nov 01 06:46:23.032564 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.032543 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" Nov 01 06:46:23.085419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.085381 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-14a6e4-openshift-default-dockercfg-vxhc4\"" Nov 01 06:46:23.101042 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.100988 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhkgq\" (UniqueName: \"kubernetes.io/projected/465cf21e-32b3-478b-9a76-99f363d61d2f-kube-api-access-lhkgq\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv\" (UID: \"465cf21e-32b3-478b-9a76-99f363d61d2f\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" Nov 01 06:46:23.101255 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.101111 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/465cf21e-32b3-478b-9a76-99f363d61d2f-workload-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv\" (UID: \"465cf21e-32b3-478b-9a76-99f363d61d2f\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" Nov 01 06:46:23.101255 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.101180 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/465cf21e-32b3-478b-9a76-99f363d61d2f-credential-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv\" (UID: \"465cf21e-32b3-478b-9a76-99f363d61d2f\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" Nov 01 06:46:23.101399 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.101309 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/465cf21e-32b3-478b-9a76-99f363d61d2f-workload-certs\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv\" (UID: \"465cf21e-32b3-478b-9a76-99f363d61d2f\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" Nov 01 06:46:23.101399 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.101354 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/465cf21e-32b3-478b-9a76-99f363d61d2f-istio-podinfo\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv\" (UID: \"465cf21e-32b3-478b-9a76-99f363d61d2f\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" Nov 01 06:46:23.101542 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.101401 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/465cf21e-32b3-478b-9a76-99f363d61d2f-istio-data\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv\" (UID: \"465cf21e-32b3-478b-9a76-99f363d61d2f\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" Nov 01 06:46:23.101542 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.101512 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/465cf21e-32b3-478b-9a76-99f363d61d2f-istiod-ca-cert\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv\" (UID: \"465cf21e-32b3-478b-9a76-99f363d61d2f\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" Nov 01 06:46:23.101663 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.101548 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/465cf21e-32b3-478b-9a76-99f363d61d2f-istio-envoy\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv\" (UID: \"465cf21e-32b3-478b-9a76-99f363d61d2f\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" Nov 01 06:46:23.101663 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.101581 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/465cf21e-32b3-478b-9a76-99f363d61d2f-istio-token\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv\" (UID: \"465cf21e-32b3-478b-9a76-99f363d61d2f\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" Nov 01 06:46:23.139967 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.139929 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv"] Nov 01 06:46:23.202538 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.202501 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/465cf21e-32b3-478b-9a76-99f363d61d2f-istio-data\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv\" (UID: \"465cf21e-32b3-478b-9a76-99f363d61d2f\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" Nov 01 06:46:23.202733 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.202543 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/465cf21e-32b3-478b-9a76-99f363d61d2f-istiod-ca-cert\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv\" (UID: \"465cf21e-32b3-478b-9a76-99f363d61d2f\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" Nov 01 06:46:23.202733 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.202578 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/465cf21e-32b3-478b-9a76-99f363d61d2f-istio-envoy\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv\" (UID: \"465cf21e-32b3-478b-9a76-99f363d61d2f\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" Nov 01 06:46:23.202733 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.202608 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/465cf21e-32b3-478b-9a76-99f363d61d2f-istio-token\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv\" (UID: \"465cf21e-32b3-478b-9a76-99f363d61d2f\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" Nov 01 06:46:23.202733 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.202688 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhkgq\" (UniqueName: \"kubernetes.io/projected/465cf21e-32b3-478b-9a76-99f363d61d2f-kube-api-access-lhkgq\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv\" (UID: \"465cf21e-32b3-478b-9a76-99f363d61d2f\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" Nov 01 06:46:23.202969 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.202736 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/465cf21e-32b3-478b-9a76-99f363d61d2f-workload-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv\" (UID: \"465cf21e-32b3-478b-9a76-99f363d61d2f\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" Nov 01 06:46:23.202969 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.202810 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/465cf21e-32b3-478b-9a76-99f363d61d2f-credential-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv\" (UID: \"465cf21e-32b3-478b-9a76-99f363d61d2f\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" Nov 01 06:46:23.202969 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.202893 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/465cf21e-32b3-478b-9a76-99f363d61d2f-workload-certs\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv\" (UID: \"465cf21e-32b3-478b-9a76-99f363d61d2f\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" Nov 01 06:46:23.202969 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.202926 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/465cf21e-32b3-478b-9a76-99f363d61d2f-istio-podinfo\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv\" (UID: \"465cf21e-32b3-478b-9a76-99f363d61d2f\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" Nov 01 06:46:23.202969 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.202963 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/465cf21e-32b3-478b-9a76-99f363d61d2f-istio-data\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv\" (UID: \"465cf21e-32b3-478b-9a76-99f363d61d2f\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" Nov 01 06:46:23.203279 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.203237 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/465cf21e-32b3-478b-9a76-99f363d61d2f-workload-certs\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv\" (UID: \"465cf21e-32b3-478b-9a76-99f363d61d2f\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" Nov 01 06:46:23.203369 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.203315 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/465cf21e-32b3-478b-9a76-99f363d61d2f-workload-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv\" (UID: \"465cf21e-32b3-478b-9a76-99f363d61d2f\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" Nov 01 06:46:23.203369 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.203333 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/465cf21e-32b3-478b-9a76-99f363d61d2f-credential-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv\" (UID: \"465cf21e-32b3-478b-9a76-99f363d61d2f\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" Nov 01 06:46:23.203730 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.203712 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/465cf21e-32b3-478b-9a76-99f363d61d2f-istiod-ca-cert\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv\" (UID: \"465cf21e-32b3-478b-9a76-99f363d61d2f\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" Nov 01 06:46:23.204992 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.204965 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/465cf21e-32b3-478b-9a76-99f363d61d2f-istio-envoy\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv\" (UID: \"465cf21e-32b3-478b-9a76-99f363d61d2f\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" Nov 01 06:46:23.205369 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.205351 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/465cf21e-32b3-478b-9a76-99f363d61d2f-istio-podinfo\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv\" (UID: \"465cf21e-32b3-478b-9a76-99f363d61d2f\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" Nov 01 06:46:23.264148 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.264114 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/465cf21e-32b3-478b-9a76-99f363d61d2f-istio-token\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv\" (UID: \"465cf21e-32b3-478b-9a76-99f363d61d2f\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" Nov 01 06:46:23.264365 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.264340 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhkgq\" (UniqueName: \"kubernetes.io/projected/465cf21e-32b3-478b-9a76-99f363d61d2f-kube-api-access-lhkgq\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv\" (UID: \"465cf21e-32b3-478b-9a76-99f363d61d2f\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" Nov 01 06:46:23.346591 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.346461 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" Nov 01 06:46:23.733393 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.733313 5040 scope.go:122] "RemoveContainer" containerID="2d6da84bdf7a22bdf8f7b77777b5ae856960206b559bebb407ab95b5ddc7b560" Nov 01 06:46:23.744389 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.744362 5040 scope.go:122] "RemoveContainer" containerID="42bcfd40bd356317d7b32b00868b4dc8104aa235bf2ccee9b7d5b02b7e3ede4a" Nov 01 06:46:23.753499 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.753466 5040 scope.go:122] "RemoveContainer" containerID="56eb0f0337dfad14bab3ec88f257a22c104667a0a88c03f4abc222e3d66dd2b0" Nov 01 06:46:23.762222 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.762200 5040 scope.go:122] "RemoveContainer" containerID="dff2bf625e6238497fae749d8f223185ff22fb93814bea246ea25f1c198800e5" Nov 01 06:46:23.777586 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.777382 5040 scope.go:122] "RemoveContainer" containerID="15d7a767d828caf0eb37f19b4e6b71c38005a986bb70883c285b08ae029878b7" Nov 01 06:46:23.778356 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.778330 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv"] Nov 01 06:46:23.780740 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:46:23.780712 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod465cf21e_32b3_478b_9a76_99f363d61d2f.slice/crio-c235546850d77e67dfb0bc6ec39c31631b164edbcd0d74cf88115f7c00157b6c WatchSource:0}: Error finding container c235546850d77e67dfb0bc6ec39c31631b164edbcd0d74cf88115f7c00157b6c: Status 404 returned error can't find the container with id c235546850d77e67dfb0bc6ec39c31631b164edbcd0d74cf88115f7c00157b6c Nov 01 06:46:23.783239 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.783205 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:46:23.783347 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.783272 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:46:23.783347 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.783302 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:46:23.859339 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:23.859299 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" event={"ID":"465cf21e-32b3-478b-9a76-99f363d61d2f","Type":"ContainerStarted","Data":"c235546850d77e67dfb0bc6ec39c31631b164edbcd0d74cf88115f7c00157b6c"} Nov 01 06:46:24.868086 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:24.868047 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" event={"ID":"465cf21e-32b3-478b-9a76-99f363d61d2f","Type":"ContainerStarted","Data":"29841af0a325ea9554082c995de40c82b9b97b8f35adc35617f8e57518d426b1"} Nov 01 06:46:25.073244 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:25.073192 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" podStartSLOduration=3.073177512 podStartE2EDuration="3.073177512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:46:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:46:25.040417688 +0000 UTC m=+3663.695534857" watchObservedRunningTime="2026-11-01 06:46:25.073177512 +0000 UTC m=+3663.728294679" Nov 01 06:46:25.347353 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:25.347310 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" Nov 01 06:46:25.349320 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:25.349291 5040 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Nov 01 06:46:25.349452 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:25.349341 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" podUID="465cf21e-32b3-478b-9a76-99f363d61d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Nov 01 06:46:26.347777 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:26.347729 5040 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Nov 01 06:46:26.348355 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:26.347798 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" podUID="465cf21e-32b3-478b-9a76-99f363d61d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Nov 01 06:46:27.347617 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:27.347582 5040 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Nov 01 06:46:27.347803 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:27.347637 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" podUID="465cf21e-32b3-478b-9a76-99f363d61d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Nov 01 06:46:28.347369 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:28.347329 5040 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Nov 01 06:46:28.347597 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:28.347391 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" podUID="465cf21e-32b3-478b-9a76-99f363d61d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Nov 01 06:46:29.347766 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:29.347715 5040 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Nov 01 06:46:29.348156 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:29.347795 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" podUID="465cf21e-32b3-478b-9a76-99f363d61d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Nov 01 06:46:30.346948 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:30.346908 5040 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Nov 01 06:46:30.347126 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:30.346978 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" podUID="465cf21e-32b3-478b-9a76-99f363d61d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Nov 01 06:46:31.763238 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:31.763196 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" Nov 01 06:46:31.763900 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:31.763866 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" Nov 01 06:46:31.764804 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:46:31.764785 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" Nov 01 06:46:39.973766 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:46:39.973723 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:47:07.052564 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:07.052522 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11a9376b-29m7k"] Nov 01 06:47:07.102936 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:07.102896 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11a9376b-29m7k"] Nov 01 06:47:07.516523 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:07.515654 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33c5b46e-9d7c-4c2b-93a2-245ff352afb3" path="/var/lib/kubelet/pods/33c5b46e-9d7c-4c2b-93a2-245ff352afb3/volumes" Nov 01 06:47:09.974542 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:47:09.974471 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:47:25.050903 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:25.050863 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7000af46-hwtsl"] Nov 01 06:47:25.100666 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:25.100631 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7000af46-hwtsl"] Nov 01 06:47:25.516100 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:25.516068 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3931ab4a-19a9-48b2-af00-ed8076b78d92" path="/var/lib/kubelet/pods/3931ab4a-19a9-48b2-af00-ed8076b78d92/volumes" Nov 01 06:47:37.049505 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:37.049404 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e1417ba-fv8rc"] Nov 01 06:47:37.099359 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:37.099324 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e1417ba-fv8rc"] Nov 01 06:47:37.517982 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:37.517191 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="309dfe7b-1534-4ba4-b376-c28cac0e9229" path="/var/lib/kubelet/pods/309dfe7b-1534-4ba4-b376-c28cac0e9229/volumes" Nov 01 06:47:39.754143 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:39.754102 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5464df48-6fstk"] Nov 01 06:47:39.758210 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:39.758191 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5464df48-6fstk" Nov 01 06:47:39.815054 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:39.815011 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5464df48-6fstk"] Nov 01 06:47:39.861565 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:39.861531 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:47:39.861565 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:39.861560 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:47:39.861783 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:39.861560 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:47:39.861783 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:39.861582 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:47:39.930830 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:39.930786 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75xrz\" (UniqueName: \"kubernetes.io/projected/50aebf01-af0d-4da8-9f2a-daee6a73880b-kube-api-access-75xrz\") pod \"maas-api-revoke-sub-5464df48-6fstk\" (UID: \"50aebf01-af0d-4da8-9f2a-daee6a73880b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5464df48-6fstk" Nov 01 06:47:39.931019 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:39.930941 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50aebf01-af0d-4da8-9f2a-daee6a73880b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5464df48-6fstk\" (UID: \"50aebf01-af0d-4da8-9f2a-daee6a73880b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5464df48-6fstk" Nov 01 06:47:39.931065 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:39.931007 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50aebf01-af0d-4da8-9f2a-daee6a73880b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5464df48-6fstk\" (UID: \"50aebf01-af0d-4da8-9f2a-daee6a73880b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5464df48-6fstk" Nov 01 06:47:39.974822 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:47:39.974782 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:47:40.032469 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:40.032378 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75xrz\" (UniqueName: \"kubernetes.io/projected/50aebf01-af0d-4da8-9f2a-daee6a73880b-kube-api-access-75xrz\") pod \"maas-api-revoke-sub-5464df48-6fstk\" (UID: \"50aebf01-af0d-4da8-9f2a-daee6a73880b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5464df48-6fstk" Nov 01 06:47:40.032656 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:40.032515 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50aebf01-af0d-4da8-9f2a-daee6a73880b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5464df48-6fstk\" (UID: \"50aebf01-af0d-4da8-9f2a-daee6a73880b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5464df48-6fstk" Nov 01 06:47:40.032656 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:40.032545 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50aebf01-af0d-4da8-9f2a-daee6a73880b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5464df48-6fstk\" (UID: \"50aebf01-af0d-4da8-9f2a-daee6a73880b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5464df48-6fstk" Nov 01 06:47:40.033240 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:40.033217 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50aebf01-af0d-4da8-9f2a-daee6a73880b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5464df48-6fstk\" (UID: \"50aebf01-af0d-4da8-9f2a-daee6a73880b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5464df48-6fstk" Nov 01 06:47:40.034874 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:40.034850 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50aebf01-af0d-4da8-9f2a-daee6a73880b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5464df48-6fstk\" (UID: \"50aebf01-af0d-4da8-9f2a-daee6a73880b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5464df48-6fstk" Nov 01 06:47:40.094091 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:40.094048 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75xrz\" (UniqueName: \"kubernetes.io/projected/50aebf01-af0d-4da8-9f2a-daee6a73880b-kube-api-access-75xrz\") pod \"maas-api-revoke-sub-5464df48-6fstk\" (UID: \"50aebf01-af0d-4da8-9f2a-daee6a73880b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5464df48-6fstk" Nov 01 06:47:40.371245 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:40.371161 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5464df48-6fstk" Nov 01 06:47:40.750720 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:40.750692 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5464df48-6fstk"] Nov 01 06:47:40.751599 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:47:40.751576 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50aebf01_af0d_4da8_9f2a_daee6a73880b.slice/crio-9db6a71071fc8df4f2472c00e787a497b27cafc92867498343d416e6adff1e8e WatchSource:0}: Error finding container 9db6a71071fc8df4f2472c00e787a497b27cafc92867498343d416e6adff1e8e: Status 404 returned error can't find the container with id 9db6a71071fc8df4f2472c00e787a497b27cafc92867498343d416e6adff1e8e Nov 01 06:47:41.412583 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:41.412460 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5464df48-6fstk" event={"ID":"50aebf01-af0d-4da8-9f2a-daee6a73880b","Type":"ContainerStarted","Data":"96fbfaa951350ee153e7e11e1dad29df612f3ae5e1e08a1f71b0636a627fccbf"} Nov 01 06:47:41.412583 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:41.412557 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5464df48-6fstk" event={"ID":"50aebf01-af0d-4da8-9f2a-daee6a73880b","Type":"ContainerStarted","Data":"9db6a71071fc8df4f2472c00e787a497b27cafc92867498343d416e6adff1e8e"} Nov 01 06:47:41.530419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:41.530355 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5464df48-6fstk" podStartSLOduration=2.530342212 podStartE2EDuration="2.530342212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:47:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:47:41.528244017 +0000 UTC m=+3740.183361186" watchObservedRunningTime="2026-11-01 06:47:41.530342212 +0000 UTC m=+3740.185459376" Nov 01 06:47:42.421026 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:42.420988 5040 generic.go:356] "Generic (PLEG): container finished" podID="50aebf01-af0d-4da8-9f2a-daee6a73880b" containerID="96fbfaa951350ee153e7e11e1dad29df612f3ae5e1e08a1f71b0636a627fccbf" exitCode=0 Nov 01 06:47:42.421568 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:42.421072 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5464df48-6fstk" event={"ID":"50aebf01-af0d-4da8-9f2a-daee6a73880b","Type":"ContainerDied","Data":"96fbfaa951350ee153e7e11e1dad29df612f3ae5e1e08a1f71b0636a627fccbf"} Nov 01 06:47:43.590913 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:43.590879 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5464df48-6fstk" Nov 01 06:47:43.779234 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:43.779201 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50aebf01-af0d-4da8-9f2a-daee6a73880b-kube-api-access-75xrz\" (UniqueName: \"kubernetes.io/projected/50aebf01-af0d-4da8-9f2a-daee6a73880b-kube-api-access-75xrz\") pod \"50aebf01-af0d-4da8-9f2a-daee6a73880b\" (UID: \"50aebf01-af0d-4da8-9f2a-daee6a73880b\") " Nov 01 06:47:43.779450 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:43.779268 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50aebf01-af0d-4da8-9f2a-daee6a73880b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50aebf01-af0d-4da8-9f2a-daee6a73880b-maas-api-openshift-ca\") pod \"50aebf01-af0d-4da8-9f2a-daee6a73880b\" (UID: \"50aebf01-af0d-4da8-9f2a-daee6a73880b\") " Nov 01 06:47:43.779450 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:43.779295 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50aebf01-af0d-4da8-9f2a-daee6a73880b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50aebf01-af0d-4da8-9f2a-daee6a73880b-maas-api-xks-ca\") pod \"50aebf01-af0d-4da8-9f2a-daee6a73880b\" (UID: \"50aebf01-af0d-4da8-9f2a-daee6a73880b\") " Nov 01 06:47:43.779681 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:43.779658 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50aebf01-af0d-4da8-9f2a-daee6a73880b-maas-api-openshift-ca" pod "50aebf01-af0d-4da8-9f2a-daee6a73880b" (UID: "50aebf01-af0d-4da8-9f2a-daee6a73880b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:47:43.781518 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:43.781467 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50aebf01-af0d-4da8-9f2a-daee6a73880b-maas-api-xks-ca" pod "50aebf01-af0d-4da8-9f2a-daee6a73880b" (UID: "50aebf01-af0d-4da8-9f2a-daee6a73880b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:47:43.781654 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:43.781583 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50aebf01-af0d-4da8-9f2a-daee6a73880b-kube-api-access-75xrz" pod "50aebf01-af0d-4da8-9f2a-daee6a73880b" (UID: "50aebf01-af0d-4da8-9f2a-daee6a73880b"). InnerVolumeSpecName "kube-api-access-75xrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:47:43.881338 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:43.881292 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75xrz\" (UniqueName: \"kubernetes.io/projected/50aebf01-af0d-4da8-9f2a-daee6a73880b-kube-api-access-75xrz\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:47:43.881338 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:43.881330 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50aebf01-af0d-4da8-9f2a-daee6a73880b-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:47:43.881619 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:43.881348 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50aebf01-af0d-4da8-9f2a-daee6a73880b-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:47:44.436262 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:44.436223 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5464df48-6fstk" Nov 01 06:47:44.436262 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:44.436238 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5464df48-6fstk" event={"ID":"50aebf01-af0d-4da8-9f2a-daee6a73880b","Type":"ContainerDied","Data":"9db6a71071fc8df4f2472c00e787a497b27cafc92867498343d416e6adff1e8e"} Nov 01 06:47:44.436522 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:44.436298 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9db6a71071fc8df4f2472c00e787a497b27cafc92867498343d416e6adff1e8e" Nov 01 06:47:50.491006 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:50.490970 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81edfced-fkjkq"] Nov 01 06:47:50.491474 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:50.491461 5040 cpu_manager.go:414] "containerMap: removing container" podUID="50aebf01-af0d-4da8-9f2a-daee6a73880b" containerName="revoke-keys" Nov 01 06:47:50.491474 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:50.491489 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50aebf01-af0d-4da8-9f2a-daee6a73880b" containerName="revoke-keys" Nov 01 06:47:50.491609 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:50.491599 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="50aebf01-af0d-4da8-9f2a-daee6a73880b" containerName="revoke-keys" Nov 01 06:47:50.495083 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:50.495067 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81edfced-fkjkq" Nov 01 06:47:50.551709 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:50.551674 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81edfced-fkjkq"] Nov 01 06:47:50.558718 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:50.558694 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/486c7011-34d6-47b7-8936-b3fc0c6fd1a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81edfced-fkjkq\" (UID: \"486c7011-34d6-47b7-8936-b3fc0c6fd1a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81edfced-fkjkq" Nov 01 06:47:50.558878 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:50.558741 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dzpf\" (UniqueName: \"kubernetes.io/projected/486c7011-34d6-47b7-8936-b3fc0c6fd1a4-kube-api-access-6dzpf\") pod \"maas-api-revoke-sub-81edfced-fkjkq\" (UID: \"486c7011-34d6-47b7-8936-b3fc0c6fd1a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81edfced-fkjkq" Nov 01 06:47:50.558878 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:50.558836 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/486c7011-34d6-47b7-8936-b3fc0c6fd1a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81edfced-fkjkq\" (UID: \"486c7011-34d6-47b7-8936-b3fc0c6fd1a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81edfced-fkjkq" Nov 01 06:47:50.598309 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:50.598275 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:47:50.598309 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:50.598282 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:47:50.598309 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:50.598286 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:47:50.598577 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:50.598448 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:47:50.660301 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:50.660267 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/486c7011-34d6-47b7-8936-b3fc0c6fd1a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81edfced-fkjkq\" (UID: \"486c7011-34d6-47b7-8936-b3fc0c6fd1a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81edfced-fkjkq" Nov 01 06:47:50.660504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:50.660330 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dzpf\" (UniqueName: \"kubernetes.io/projected/486c7011-34d6-47b7-8936-b3fc0c6fd1a4-kube-api-access-6dzpf\") pod \"maas-api-revoke-sub-81edfced-fkjkq\" (UID: \"486c7011-34d6-47b7-8936-b3fc0c6fd1a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81edfced-fkjkq" Nov 01 06:47:50.660504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:50.660371 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/486c7011-34d6-47b7-8936-b3fc0c6fd1a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81edfced-fkjkq\" (UID: \"486c7011-34d6-47b7-8936-b3fc0c6fd1a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81edfced-fkjkq" Nov 01 06:47:50.661101 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:50.661081 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/486c7011-34d6-47b7-8936-b3fc0c6fd1a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81edfced-fkjkq\" (UID: \"486c7011-34d6-47b7-8936-b3fc0c6fd1a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81edfced-fkjkq" Nov 01 06:47:50.662852 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:50.662820 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/486c7011-34d6-47b7-8936-b3fc0c6fd1a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81edfced-fkjkq\" (UID: \"486c7011-34d6-47b7-8936-b3fc0c6fd1a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81edfced-fkjkq" Nov 01 06:47:50.721636 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:50.721597 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dzpf\" (UniqueName: \"kubernetes.io/projected/486c7011-34d6-47b7-8936-b3fc0c6fd1a4-kube-api-access-6dzpf\") pod \"maas-api-revoke-sub-81edfced-fkjkq\" (UID: \"486c7011-34d6-47b7-8936-b3fc0c6fd1a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81edfced-fkjkq" Nov 01 06:47:50.810425 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:50.810385 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81edfced-fkjkq" Nov 01 06:47:51.186405 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:51.186377 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81edfced-fkjkq"] Nov 01 06:47:51.187530 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:47:51.187473 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod486c7011_34d6_47b7_8936_b3fc0c6fd1a4.slice/crio-4320841081c06604da6418b41deafc89994b3f2218f7c937af7df60f4bcc6e1c WatchSource:0}: Error finding container 4320841081c06604da6418b41deafc89994b3f2218f7c937af7df60f4bcc6e1c: Status 404 returned error can't find the container with id 4320841081c06604da6418b41deafc89994b3f2218f7c937af7df60f4bcc6e1c Nov 01 06:47:51.486526 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:51.486410 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81edfced-fkjkq" event={"ID":"486c7011-34d6-47b7-8936-b3fc0c6fd1a4","Type":"ContainerStarted","Data":"4320841081c06604da6418b41deafc89994b3f2218f7c937af7df60f4bcc6e1c"} Nov 01 06:47:52.495171 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:52.495080 5040 generic.go:356] "Generic (PLEG): container finished" podID="486c7011-34d6-47b7-8936-b3fc0c6fd1a4" containerID="88436aa3a5125f8c79098479d5b4f0f325c459d4d7de7711f8a361e87b259698" exitCode=0 Nov 01 06:47:52.495567 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:52.495168 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81edfced-fkjkq" event={"ID":"486c7011-34d6-47b7-8936-b3fc0c6fd1a4","Type":"ContainerDied","Data":"88436aa3a5125f8c79098479d5b4f0f325c459d4d7de7711f8a361e87b259698"} Nov 01 06:47:53.667436 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:53.667407 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81edfced-fkjkq" Nov 01 06:47:53.696462 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:53.696430 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/486c7011-34d6-47b7-8936-b3fc0c6fd1a4-kube-api-access-6dzpf\" (UniqueName: \"kubernetes.io/projected/486c7011-34d6-47b7-8936-b3fc0c6fd1a4-kube-api-access-6dzpf\") pod \"486c7011-34d6-47b7-8936-b3fc0c6fd1a4\" (UID: \"486c7011-34d6-47b7-8936-b3fc0c6fd1a4\") " Nov 01 06:47:53.696650 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:53.696471 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/486c7011-34d6-47b7-8936-b3fc0c6fd1a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/486c7011-34d6-47b7-8936-b3fc0c6fd1a4-maas-api-openshift-ca\") pod \"486c7011-34d6-47b7-8936-b3fc0c6fd1a4\" (UID: \"486c7011-34d6-47b7-8936-b3fc0c6fd1a4\") " Nov 01 06:47:53.696650 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:53.696612 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/486c7011-34d6-47b7-8936-b3fc0c6fd1a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/486c7011-34d6-47b7-8936-b3fc0c6fd1a4-maas-api-xks-ca\") pod \"486c7011-34d6-47b7-8936-b3fc0c6fd1a4\" (UID: \"486c7011-34d6-47b7-8936-b3fc0c6fd1a4\") " Nov 01 06:47:53.696869 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:53.696845 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/486c7011-34d6-47b7-8936-b3fc0c6fd1a4-maas-api-openshift-ca" pod "486c7011-34d6-47b7-8936-b3fc0c6fd1a4" (UID: "486c7011-34d6-47b7-8936-b3fc0c6fd1a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:47:53.697049 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:53.697031 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/486c7011-34d6-47b7-8936-b3fc0c6fd1a4-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:47:53.698683 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:53.698657 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/486c7011-34d6-47b7-8936-b3fc0c6fd1a4-maas-api-xks-ca" pod "486c7011-34d6-47b7-8936-b3fc0c6fd1a4" (UID: "486c7011-34d6-47b7-8936-b3fc0c6fd1a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:47:53.698788 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:53.698687 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/486c7011-34d6-47b7-8936-b3fc0c6fd1a4-kube-api-access-6dzpf" pod "486c7011-34d6-47b7-8936-b3fc0c6fd1a4" (UID: "486c7011-34d6-47b7-8936-b3fc0c6fd1a4"). InnerVolumeSpecName "kube-api-access-6dzpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:47:53.797836 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:53.797751 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dzpf\" (UniqueName: \"kubernetes.io/projected/486c7011-34d6-47b7-8936-b3fc0c6fd1a4-kube-api-access-6dzpf\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:47:53.797836 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:53.797782 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/486c7011-34d6-47b7-8936-b3fc0c6fd1a4-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:47:54.511625 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:54.511586 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81edfced-fkjkq" Nov 01 06:47:54.511823 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:54.511621 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81edfced-fkjkq" event={"ID":"486c7011-34d6-47b7-8936-b3fc0c6fd1a4","Type":"ContainerDied","Data":"4320841081c06604da6418b41deafc89994b3f2218f7c937af7df60f4bcc6e1c"} Nov 01 06:47:54.511823 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:47:54.511665 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4320841081c06604da6418b41deafc89994b3f2218f7c937af7df60f4bcc6e1c" Nov 01 06:48:09.048055 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:09.048013 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2932d07-rwmhq"] Nov 01 06:48:09.098294 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:09.098247 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2932d07-rwmhq"] Nov 01 06:48:09.255006 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:09.254967 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8"] Nov 01 06:48:09.255343 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:09.255322 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x"] Nov 01 06:48:09.255742 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:09.255688 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" podUID="0f4f1372-1a38-4d0b-b73b-9a9fa8b881de" containerName="istio-proxy" containerID="cri-o://20708662581ab371ed8b56bb6d07bff07a8f8d2a5dcf7ef260ea6f26dfb987e3" gracePeriod=30 Nov 01 06:48:09.256259 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:09.256175 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" podUID="b6760e57-3c38-4dda-84e0-75b99b3a70a6" containerName="istio-proxy" containerID="cri-o://4804258cb3e72067ce2e5d304ec340f0c2267f9beef737db9b108dbe00fa85ec" gracePeriod=30 Nov 01 06:48:09.297882 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:09.297843 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl"] Nov 01 06:48:09.298134 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:09.298076 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" podUID="b0ce9454-b676-4e7d-abd3-34d76aa2c787" containerName="istio-proxy" containerID="cri-o://de1372a895791e68004c9e81f9d24fd2773533ae800a18e317124abe98f8caff" gracePeriod=30 Nov 01 06:48:09.301297 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:09.301266 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj"] Nov 01 06:48:09.301690 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:09.301639 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" podUID="9e6f829a-4773-4740-a9e4-328091ee920f" containerName="istio-proxy" containerID="cri-o://ae843ba1e8e079961f95d588c37ea16c76aeb714ead9448c841153ddbb0f57c0" gracePeriod=30 Nov 01 06:48:09.515458 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:09.515420 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c0a1651-82be-44b2-93b2-b9130f494cba" path="/var/lib/kubelet/pods/4c0a1651-82be-44b2-93b2-b9130f494cba/volumes" Nov 01 06:48:09.973804 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:48:09.973769 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:48:11.843577 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:11.843539 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d67d7f00-xmj9q"] Nov 01 06:48:11.844117 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:11.844031 5040 cpu_manager.go:414] "containerMap: removing container" podUID="486c7011-34d6-47b7-8936-b3fc0c6fd1a4" containerName="revoke-keys" Nov 01 06:48:11.844117 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:11.844044 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="486c7011-34d6-47b7-8936-b3fc0c6fd1a4" containerName="revoke-keys" Nov 01 06:48:11.844196 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:11.844158 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="486c7011-34d6-47b7-8936-b3fc0c6fd1a4" containerName="revoke-keys" Nov 01 06:48:11.847635 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:11.847617 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67d7f00-xmj9q" Nov 01 06:48:11.900775 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:11.900740 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:48:11.951115 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:11.951082 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:48:11.951307 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:11.951162 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:48:11.951307 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:11.951163 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:48:11.955852 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:11.955821 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d67d7f00-xmj9q"] Nov 01 06:48:12.006205 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:12.006170 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb479226-2d4d-470f-9202-2ca474ad9f5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d67d7f00-xmj9q\" (UID: \"fb479226-2d4d-470f-9202-2ca474ad9f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67d7f00-xmj9q" Nov 01 06:48:12.006386 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:12.006211 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb479226-2d4d-470f-9202-2ca474ad9f5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d67d7f00-xmj9q\" (UID: \"fb479226-2d4d-470f-9202-2ca474ad9f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67d7f00-xmj9q" Nov 01 06:48:12.006430 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:12.006397 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kbb7\" (UniqueName: \"kubernetes.io/projected/fb479226-2d4d-470f-9202-2ca474ad9f5c-kube-api-access-4kbb7\") pod \"maas-api-revoke-sub-d67d7f00-xmj9q\" (UID: \"fb479226-2d4d-470f-9202-2ca474ad9f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67d7f00-xmj9q" Nov 01 06:48:12.108068 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:12.107983 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb479226-2d4d-470f-9202-2ca474ad9f5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d67d7f00-xmj9q\" (UID: \"fb479226-2d4d-470f-9202-2ca474ad9f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67d7f00-xmj9q" Nov 01 06:48:12.108068 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:12.108028 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb479226-2d4d-470f-9202-2ca474ad9f5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d67d7f00-xmj9q\" (UID: \"fb479226-2d4d-470f-9202-2ca474ad9f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67d7f00-xmj9q" Nov 01 06:48:12.108266 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:12.108135 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kbb7\" (UniqueName: \"kubernetes.io/projected/fb479226-2d4d-470f-9202-2ca474ad9f5c-kube-api-access-4kbb7\") pod \"maas-api-revoke-sub-d67d7f00-xmj9q\" (UID: \"fb479226-2d4d-470f-9202-2ca474ad9f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67d7f00-xmj9q" Nov 01 06:48:12.108803 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:12.108776 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb479226-2d4d-470f-9202-2ca474ad9f5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d67d7f00-xmj9q\" (UID: \"fb479226-2d4d-470f-9202-2ca474ad9f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67d7f00-xmj9q" Nov 01 06:48:12.110306 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:12.110280 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb479226-2d4d-470f-9202-2ca474ad9f5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d67d7f00-xmj9q\" (UID: \"fb479226-2d4d-470f-9202-2ca474ad9f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67d7f00-xmj9q" Nov 01 06:48:12.170216 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:12.170181 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kbb7\" (UniqueName: \"kubernetes.io/projected/fb479226-2d4d-470f-9202-2ca474ad9f5c-kube-api-access-4kbb7\") pod \"maas-api-revoke-sub-d67d7f00-xmj9q\" (UID: \"fb479226-2d4d-470f-9202-2ca474ad9f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67d7f00-xmj9q" Nov 01 06:48:12.461626 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:12.461548 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67d7f00-xmj9q" Nov 01 06:48:12.844472 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:12.844427 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d67d7f00-xmj9q"] Nov 01 06:48:12.844838 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:48:12.844685 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb479226_2d4d_470f_9202_2ca474ad9f5c.slice/crio-bc8fdc6a09e62a03e51e0ed60047a6a5d32b3bb186cf876a252bdb803c5fbf94 WatchSource:0}: Error finding container bc8fdc6a09e62a03e51e0ed60047a6a5d32b3bb186cf876a252bdb803c5fbf94: Status 404 returned error can't find the container with id bc8fdc6a09e62a03e51e0ed60047a6a5d32b3bb186cf876a252bdb803c5fbf94 Nov 01 06:48:13.643662 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:13.643624 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67d7f00-xmj9q" event={"ID":"fb479226-2d4d-470f-9202-2ca474ad9f5c","Type":"ContainerStarted","Data":"ad7e9a907e506659e1dd815583c5764e28958ecca56699633903b1b20a2ccac9"} Nov 01 06:48:13.643846 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:13.643685 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67d7f00-xmj9q" event={"ID":"fb479226-2d4d-470f-9202-2ca474ad9f5c","Type":"ContainerStarted","Data":"bc8fdc6a09e62a03e51e0ed60047a6a5d32b3bb186cf876a252bdb803c5fbf94"} Nov 01 06:48:13.761696 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:13.761641 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67d7f00-xmj9q" podStartSLOduration=2.76162669 podStartE2EDuration="2.76162669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:48:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:48:13.760371496 +0000 UTC m=+3772.415488645" watchObservedRunningTime="2026-11-01 06:48:13.76162669 +0000 UTC m=+3772.416743857" Nov 01 06:48:14.597924 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.597890 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" Nov 01 06:48:14.629535 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.629506 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" Nov 01 06:48:14.655457 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.655417 5040 generic.go:356] "Generic (PLEG): container finished" podID="fb479226-2d4d-470f-9202-2ca474ad9f5c" containerID="ad7e9a907e506659e1dd815583c5764e28958ecca56699633903b1b20a2ccac9" exitCode=0 Nov 01 06:48:14.655676 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.655526 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67d7f00-xmj9q" event={"ID":"fb479226-2d4d-470f-9202-2ca474ad9f5c","Type":"ContainerDied","Data":"ad7e9a907e506659e1dd815583c5764e28958ecca56699633903b1b20a2ccac9"} Nov 01 06:48:14.657106 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.657075 5040 generic.go:356] "Generic (PLEG): container finished" podID="b0ce9454-b676-4e7d-abd3-34d76aa2c787" containerID="de1372a895791e68004c9e81f9d24fd2773533ae800a18e317124abe98f8caff" exitCode=0 Nov 01 06:48:14.657246 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.657118 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" Nov 01 06:48:14.657246 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.657154 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" event={"ID":"b0ce9454-b676-4e7d-abd3-34d76aa2c787","Type":"ContainerDied","Data":"de1372a895791e68004c9e81f9d24fd2773533ae800a18e317124abe98f8caff"} Nov 01 06:48:14.657246 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.657212 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl" event={"ID":"b0ce9454-b676-4e7d-abd3-34d76aa2c787","Type":"ContainerDied","Data":"1441df6a726d78d3f626d4f47384efdec36ede31741937f048ecee4038691f88"} Nov 01 06:48:14.657432 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.657246 5040 scope.go:122] "RemoveContainer" containerID="de1372a895791e68004c9e81f9d24fd2773533ae800a18e317124abe98f8caff" Nov 01 06:48:14.659216 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.659191 5040 generic.go:356] "Generic (PLEG): container finished" podID="0f4f1372-1a38-4d0b-b73b-9a9fa8b881de" containerID="20708662581ab371ed8b56bb6d07bff07a8f8d2a5dcf7ef260ea6f26dfb987e3" exitCode=0 Nov 01 06:48:14.659322 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.659293 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" event={"ID":"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de","Type":"ContainerDied","Data":"20708662581ab371ed8b56bb6d07bff07a8f8d2a5dcf7ef260ea6f26dfb987e3"} Nov 01 06:48:14.660965 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.660942 5040 generic.go:356] "Generic (PLEG): container finished" podID="b6760e57-3c38-4dda-84e0-75b99b3a70a6" containerID="4804258cb3e72067ce2e5d304ec340f0c2267f9beef737db9b108dbe00fa85ec" exitCode=0 Nov 01 06:48:14.661082 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.661011 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" event={"ID":"b6760e57-3c38-4dda-84e0-75b99b3a70a6","Type":"ContainerDied","Data":"4804258cb3e72067ce2e5d304ec340f0c2267f9beef737db9b108dbe00fa85ec"} Nov 01 06:48:14.661082 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.661039 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" event={"ID":"b6760e57-3c38-4dda-84e0-75b99b3a70a6","Type":"ContainerDied","Data":"99ef2e27d9e89140578cbe09df8098905f85fcfd9d0f4dfc7c5b4ecc81399a3e"} Nov 01 06:48:14.661209 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.661088 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8" Nov 01 06:48:14.664045 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.664021 5040 generic.go:356] "Generic (PLEG): container finished" podID="9e6f829a-4773-4740-a9e4-328091ee920f" containerID="ae843ba1e8e079961f95d588c37ea16c76aeb714ead9448c841153ddbb0f57c0" exitCode=0 Nov 01 06:48:14.664146 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.664091 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" event={"ID":"9e6f829a-4773-4740-a9e4-328091ee920f","Type":"ContainerDied","Data":"ae843ba1e8e079961f95d588c37ea16c76aeb714ead9448c841153ddbb0f57c0"} Nov 01 06:48:14.669663 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.669641 5040 scope.go:122] "RemoveContainer" containerID="de1372a895791e68004c9e81f9d24fd2773533ae800a18e317124abe98f8caff" Nov 01 06:48:14.670039 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:48:14.670000 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de1372a895791e68004c9e81f9d24fd2773533ae800a18e317124abe98f8caff\": container with ID starting with de1372a895791e68004c9e81f9d24fd2773533ae800a18e317124abe98f8caff not found: ID does not exist" containerID="de1372a895791e68004c9e81f9d24fd2773533ae800a18e317124abe98f8caff" Nov 01 06:48:14.670126 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.670052 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de1372a895791e68004c9e81f9d24fd2773533ae800a18e317124abe98f8caff"} err="failed to get container status \"de1372a895791e68004c9e81f9d24fd2773533ae800a18e317124abe98f8caff\": rpc error: code = NotFound desc = could not find container \"de1372a895791e68004c9e81f9d24fd2773533ae800a18e317124abe98f8caff\": container with ID starting with de1372a895791e68004c9e81f9d24fd2773533ae800a18e317124abe98f8caff not found: ID does not exist" Nov 01 06:48:14.670126 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.670076 5040 scope.go:122] "RemoveContainer" containerID="4804258cb3e72067ce2e5d304ec340f0c2267f9beef737db9b108dbe00fa85ec" Nov 01 06:48:14.681633 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.681611 5040 scope.go:122] "RemoveContainer" containerID="4804258cb3e72067ce2e5d304ec340f0c2267f9beef737db9b108dbe00fa85ec" Nov 01 06:48:14.682116 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:48:14.682096 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4804258cb3e72067ce2e5d304ec340f0c2267f9beef737db9b108dbe00fa85ec\": container with ID starting with 4804258cb3e72067ce2e5d304ec340f0c2267f9beef737db9b108dbe00fa85ec not found: ID does not exist" containerID="4804258cb3e72067ce2e5d304ec340f0c2267f9beef737db9b108dbe00fa85ec" Nov 01 06:48:14.682177 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.682117 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" Nov 01 06:48:14.682177 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.682126 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4804258cb3e72067ce2e5d304ec340f0c2267f9beef737db9b108dbe00fa85ec"} err="failed to get container status \"4804258cb3e72067ce2e5d304ec340f0c2267f9beef737db9b108dbe00fa85ec\": rpc error: code = NotFound desc = could not find container \"4804258cb3e72067ce2e5d304ec340f0c2267f9beef737db9b108dbe00fa85ec\": container with ID starting with 4804258cb3e72067ce2e5d304ec340f0c2267f9beef737db9b108dbe00fa85ec not found: ID does not exist" Nov 01 06:48:14.690547 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.690521 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" Nov 01 06:48:14.737020 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.736976 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6760e57-3c38-4dda-84e0-75b99b3a70a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6760e57-3c38-4dda-84e0-75b99b3a70a6-istiod-ca-cert\") pod \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\" (UID: \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\") " Nov 01 06:48:14.737020 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.737021 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0ce9454-b676-4e7d-abd3-34d76aa2c787-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0ce9454-b676-4e7d-abd3-34d76aa2c787-istiod-ca-cert\") pod \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\" (UID: \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\") " Nov 01 06:48:14.737271 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.737072 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0ce9454-b676-4e7d-abd3-34d76aa2c787-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0ce9454-b676-4e7d-abd3-34d76aa2c787-istio-data\") pod \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\" (UID: \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\") " Nov 01 06:48:14.737271 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.737092 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0ce9454-b676-4e7d-abd3-34d76aa2c787-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ce9454-b676-4e7d-abd3-34d76aa2c787-credential-socket\") pod \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\" (UID: \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\") " Nov 01 06:48:14.737271 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.737114 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6760e57-3c38-4dda-84e0-75b99b3a70a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6760e57-3c38-4dda-84e0-75b99b3a70a6-istio-envoy\") pod \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\" (UID: \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\") " Nov 01 06:48:14.737271 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.737152 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6760e57-3c38-4dda-84e0-75b99b3a70a6-kube-api-access-jtz6r\" (UniqueName: \"kubernetes.io/projected/b6760e57-3c38-4dda-84e0-75b99b3a70a6-kube-api-access-jtz6r\") pod \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\" (UID: \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\") " Nov 01 06:48:14.737271 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.737186 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6760e57-3c38-4dda-84e0-75b99b3a70a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6760e57-3c38-4dda-84e0-75b99b3a70a6-istio-podinfo\") pod \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\" (UID: \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\") " Nov 01 06:48:14.737271 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.737237 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0ce9454-b676-4e7d-abd3-34d76aa2c787-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ce9454-b676-4e7d-abd3-34d76aa2c787-workload-socket\") pod \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\" (UID: \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\") " Nov 01 06:48:14.737271 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.737271 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0ce9454-b676-4e7d-abd3-34d76aa2c787-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0ce9454-b676-4e7d-abd3-34d76aa2c787-istio-envoy\") pod \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\" (UID: \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\") " Nov 01 06:48:14.737674 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.737299 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0ce9454-b676-4e7d-abd3-34d76aa2c787-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0ce9454-b676-4e7d-abd3-34d76aa2c787-workload-certs\") pod \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\" (UID: \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\") " Nov 01 06:48:14.737674 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.737337 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6760e57-3c38-4dda-84e0-75b99b3a70a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6760e57-3c38-4dda-84e0-75b99b3a70a6-workload-certs\") pod \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\" (UID: \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\") " Nov 01 06:48:14.737674 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.737372 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0ce9454-b676-4e7d-abd3-34d76aa2c787-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0ce9454-b676-4e7d-abd3-34d76aa2c787-istio-podinfo\") pod \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\" (UID: \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\") " Nov 01 06:48:14.737674 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.737404 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0ce9454-b676-4e7d-abd3-34d76aa2c787-istio-data" pod "b0ce9454-b676-4e7d-abd3-34d76aa2c787" (UID: "b0ce9454-b676-4e7d-abd3-34d76aa2c787"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:48:14.737674 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.737423 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0ce9454-b676-4e7d-abd3-34d76aa2c787-credential-socket" pod "b0ce9454-b676-4e7d-abd3-34d76aa2c787" (UID: "b0ce9454-b676-4e7d-abd3-34d76aa2c787"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:48:14.737674 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.737443 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0ce9454-b676-4e7d-abd3-34d76aa2c787-kube-api-access-ln9vr\" (UniqueName: \"kubernetes.io/projected/b0ce9454-b676-4e7d-abd3-34d76aa2c787-kube-api-access-ln9vr\") pod \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\" (UID: \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\") " Nov 01 06:48:14.737674 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.737456 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6760e57-3c38-4dda-84e0-75b99b3a70a6-istiod-ca-cert" pod "b6760e57-3c38-4dda-84e0-75b99b3a70a6" (UID: "b6760e57-3c38-4dda-84e0-75b99b3a70a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:48:14.737674 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.737524 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6760e57-3c38-4dda-84e0-75b99b3a70a6-istio-token\" (UniqueName: \"kubernetes.io/projected/b6760e57-3c38-4dda-84e0-75b99b3a70a6-istio-token\") pod \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\" (UID: \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\") " Nov 01 06:48:14.737674 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.737568 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6760e57-3c38-4dda-84e0-75b99b3a70a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6760e57-3c38-4dda-84e0-75b99b3a70a6-credential-socket\") pod \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\" (UID: \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\") " Nov 01 06:48:14.737674 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.737596 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6760e57-3c38-4dda-84e0-75b99b3a70a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6760e57-3c38-4dda-84e0-75b99b3a70a6-workload-socket\") pod \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\" (UID: \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\") " Nov 01 06:48:14.737674 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.737633 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0ce9454-b676-4e7d-abd3-34d76aa2c787-istio-token\" (UniqueName: \"kubernetes.io/projected/b0ce9454-b676-4e7d-abd3-34d76aa2c787-istio-token\") pod \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\" (UID: \"b0ce9454-b676-4e7d-abd3-34d76aa2c787\") " Nov 01 06:48:14.737674 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.737666 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6760e57-3c38-4dda-84e0-75b99b3a70a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6760e57-3c38-4dda-84e0-75b99b3a70a6-istio-data\") pod \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\" (UID: \"b6760e57-3c38-4dda-84e0-75b99b3a70a6\") " Nov 01 06:48:14.737674 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.737673 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6760e57-3c38-4dda-84e0-75b99b3a70a6-workload-certs" pod "b6760e57-3c38-4dda-84e0-75b99b3a70a6" (UID: "b6760e57-3c38-4dda-84e0-75b99b3a70a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:48:14.738411 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.737691 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0ce9454-b676-4e7d-abd3-34d76aa2c787-workload-certs" pod "b0ce9454-b676-4e7d-abd3-34d76aa2c787" (UID: "b0ce9454-b676-4e7d-abd3-34d76aa2c787"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:48:14.738411 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.737888 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0ce9454-b676-4e7d-abd3-34d76aa2c787-workload-socket" pod "b0ce9454-b676-4e7d-abd3-34d76aa2c787" (UID: "b0ce9454-b676-4e7d-abd3-34d76aa2c787"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:48:14.738411 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.737964 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0ce9454-b676-4e7d-abd3-34d76aa2c787-istiod-ca-cert" pod "b0ce9454-b676-4e7d-abd3-34d76aa2c787" (UID: "b0ce9454-b676-4e7d-abd3-34d76aa2c787"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:48:14.738411 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.738117 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6760e57-3c38-4dda-84e0-75b99b3a70a6-istio-data" pod "b6760e57-3c38-4dda-84e0-75b99b3a70a6" (UID: "b6760e57-3c38-4dda-84e0-75b99b3a70a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:48:14.738411 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.738231 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6760e57-3c38-4dda-84e0-75b99b3a70a6-credential-socket" pod "b6760e57-3c38-4dda-84e0-75b99b3a70a6" (UID: "b6760e57-3c38-4dda-84e0-75b99b3a70a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:48:14.738411 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.738328 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6760e57-3c38-4dda-84e0-75b99b3a70a6-workload-socket" pod "b6760e57-3c38-4dda-84e0-75b99b3a70a6" (UID: "b6760e57-3c38-4dda-84e0-75b99b3a70a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:48:14.738411 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.738353 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6760e57-3c38-4dda-84e0-75b99b3a70a6-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:14.738411 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.738377 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0ce9454-b676-4e7d-abd3-34d76aa2c787-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:14.738411 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.738394 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0ce9454-b676-4e7d-abd3-34d76aa2c787-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:14.738411 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.738409 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ce9454-b676-4e7d-abd3-34d76aa2c787-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:14.739030 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.738424 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ce9454-b676-4e7d-abd3-34d76aa2c787-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:14.739030 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.738439 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0ce9454-b676-4e7d-abd3-34d76aa2c787-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:14.739030 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.738453 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6760e57-3c38-4dda-84e0-75b99b3a70a6-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:14.739030 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.738468 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6760e57-3c38-4dda-84e0-75b99b3a70a6-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:14.739030 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.738504 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6760e57-3c38-4dda-84e0-75b99b3a70a6-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:14.740303 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.740261 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0ce9454-b676-4e7d-abd3-34d76aa2c787-istio-podinfo" pod "b0ce9454-b676-4e7d-abd3-34d76aa2c787" (UID: "b0ce9454-b676-4e7d-abd3-34d76aa2c787"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:48:14.740438 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.740404 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6760e57-3c38-4dda-84e0-75b99b3a70a6-kube-api-access-jtz6r" pod "b6760e57-3c38-4dda-84e0-75b99b3a70a6" (UID: "b6760e57-3c38-4dda-84e0-75b99b3a70a6"). InnerVolumeSpecName "kube-api-access-jtz6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:48:14.740719 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.740695 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ce9454-b676-4e7d-abd3-34d76aa2c787-istio-token" pod "b0ce9454-b676-4e7d-abd3-34d76aa2c787" (UID: "b0ce9454-b676-4e7d-abd3-34d76aa2c787"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:48:14.740845 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.740826 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0ce9454-b676-4e7d-abd3-34d76aa2c787-istio-envoy" pod "b0ce9454-b676-4e7d-abd3-34d76aa2c787" (UID: "b0ce9454-b676-4e7d-abd3-34d76aa2c787"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:48:14.740941 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.740921 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6760e57-3c38-4dda-84e0-75b99b3a70a6-istio-token" pod "b6760e57-3c38-4dda-84e0-75b99b3a70a6" (UID: "b6760e57-3c38-4dda-84e0-75b99b3a70a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:48:14.741093 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.741067 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ce9454-b676-4e7d-abd3-34d76aa2c787-kube-api-access-ln9vr" pod "b0ce9454-b676-4e7d-abd3-34d76aa2c787" (UID: "b0ce9454-b676-4e7d-abd3-34d76aa2c787"). InnerVolumeSpecName "kube-api-access-ln9vr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:48:14.741093 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.741075 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6760e57-3c38-4dda-84e0-75b99b3a70a6-istio-envoy" pod "b6760e57-3c38-4dda-84e0-75b99b3a70a6" (UID: "b6760e57-3c38-4dda-84e0-75b99b3a70a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:48:14.741219 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.741067 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6760e57-3c38-4dda-84e0-75b99b3a70a6-istio-podinfo" pod "b6760e57-3c38-4dda-84e0-75b99b3a70a6" (UID: "b6760e57-3c38-4dda-84e0-75b99b3a70a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:48:14.839766 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.839685 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e6f829a-4773-4740-a9e4-328091ee920f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e6f829a-4773-4740-a9e4-328091ee920f-istiod-ca-cert\") pod \"9e6f829a-4773-4740-a9e4-328091ee920f\" (UID: \"9e6f829a-4773-4740-a9e4-328091ee920f\") " Nov 01 06:48:14.839766 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.839730 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e6f829a-4773-4740-a9e4-328091ee920f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e6f829a-4773-4740-a9e4-328091ee920f-workload-socket\") pod \"9e6f829a-4773-4740-a9e4-328091ee920f\" (UID: \"9e6f829a-4773-4740-a9e4-328091ee920f\") " Nov 01 06:48:14.839766 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.839767 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e6f829a-4773-4740-a9e4-328091ee920f-kube-api-access-tl257\" (UniqueName: \"kubernetes.io/projected/9e6f829a-4773-4740-a9e4-328091ee920f-kube-api-access-tl257\") pod \"9e6f829a-4773-4740-a9e4-328091ee920f\" (UID: \"9e6f829a-4773-4740-a9e4-328091ee920f\") " Nov 01 06:48:14.840064 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.839801 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-istio-podinfo\") pod \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\" (UID: \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\") " Nov 01 06:48:14.840064 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.839832 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e6f829a-4773-4740-a9e4-328091ee920f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e6f829a-4773-4740-a9e4-328091ee920f-credential-socket\") pod \"9e6f829a-4773-4740-a9e4-328091ee920f\" (UID: \"9e6f829a-4773-4740-a9e4-328091ee920f\") " Nov 01 06:48:14.840064 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.839864 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-workload-socket\") pod \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\" (UID: \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\") " Nov 01 06:48:14.840064 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.839892 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-credential-socket\") pod \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\" (UID: \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\") " Nov 01 06:48:14.840064 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.839931 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-workload-certs\") pod \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\" (UID: \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\") " Nov 01 06:48:14.840064 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.839989 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-istio-token\" (UniqueName: \"kubernetes.io/projected/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-istio-token\") pod \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\" (UID: \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\") " Nov 01 06:48:14.840064 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.840017 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e6f829a-4773-4740-a9e4-328091ee920f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e6f829a-4773-4740-a9e4-328091ee920f-istio-data\") pod \"9e6f829a-4773-4740-a9e4-328091ee920f\" (UID: \"9e6f829a-4773-4740-a9e4-328091ee920f\") " Nov 01 06:48:14.840473 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.840086 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e6f829a-4773-4740-a9e4-328091ee920f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e6f829a-4773-4740-a9e4-328091ee920f-workload-certs\") pod \"9e6f829a-4773-4740-a9e4-328091ee920f\" (UID: \"9e6f829a-4773-4740-a9e4-328091ee920f\") " Nov 01 06:48:14.840473 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.840126 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-istiod-ca-cert\") pod \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\" (UID: \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\") " Nov 01 06:48:14.840473 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.840139 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e6f829a-4773-4740-a9e4-328091ee920f-workload-socket" pod "9e6f829a-4773-4740-a9e4-328091ee920f" (UID: "9e6f829a-4773-4740-a9e4-328091ee920f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:48:14.840473 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.840165 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e6f829a-4773-4740-a9e4-328091ee920f-istio-token\" (UniqueName: \"kubernetes.io/projected/9e6f829a-4773-4740-a9e4-328091ee920f-istio-token\") pod \"9e6f829a-4773-4740-a9e4-328091ee920f\" (UID: \"9e6f829a-4773-4740-a9e4-328091ee920f\") " Nov 01 06:48:14.840473 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.840192 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e6f829a-4773-4740-a9e4-328091ee920f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e6f829a-4773-4740-a9e4-328091ee920f-istio-envoy\") pod \"9e6f829a-4773-4740-a9e4-328091ee920f\" (UID: \"9e6f829a-4773-4740-a9e4-328091ee920f\") " Nov 01 06:48:14.840473 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.840149 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-workload-socket" pod "0f4f1372-1a38-4d0b-b73b-9a9fa8b881de" (UID: "0f4f1372-1a38-4d0b-b73b-9a9fa8b881de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:48:14.840473 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.840222 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e6f829a-4773-4740-a9e4-328091ee920f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e6f829a-4773-4740-a9e4-328091ee920f-istio-podinfo\") pod \"9e6f829a-4773-4740-a9e4-328091ee920f\" (UID: \"9e6f829a-4773-4740-a9e4-328091ee920f\") " Nov 01 06:48:14.840473 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.840228 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e6f829a-4773-4740-a9e4-328091ee920f-credential-socket" pod "9e6f829a-4773-4740-a9e4-328091ee920f" (UID: "9e6f829a-4773-4740-a9e4-328091ee920f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:48:14.840473 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.840162 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e6f829a-4773-4740-a9e4-328091ee920f-istiod-ca-cert" pod "9e6f829a-4773-4740-a9e4-328091ee920f" (UID: "9e6f829a-4773-4740-a9e4-328091ee920f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:48:14.840473 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.840289 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-istio-envoy\") pod \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\" (UID: \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\") " Nov 01 06:48:14.840473 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.840310 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-workload-certs" pod "0f4f1372-1a38-4d0b-b73b-9a9fa8b881de" (UID: "0f4f1372-1a38-4d0b-b73b-9a9fa8b881de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:48:14.840473 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.840359 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-kube-api-access-6d9xk\" (UniqueName: \"kubernetes.io/projected/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-kube-api-access-6d9xk\") pod \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\" (UID: \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\") " Nov 01 06:48:14.840473 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.840401 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e6f829a-4773-4740-a9e4-328091ee920f-workload-certs" pod "9e6f829a-4773-4740-a9e4-328091ee920f" (UID: "9e6f829a-4773-4740-a9e4-328091ee920f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:48:14.840473 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.840409 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-istio-data\") pod \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\" (UID: \"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de\") " Nov 01 06:48:14.841281 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.840565 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-credential-socket" pod "0f4f1372-1a38-4d0b-b73b-9a9fa8b881de" (UID: "0f4f1372-1a38-4d0b-b73b-9a9fa8b881de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:48:14.841281 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.840645 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e6f829a-4773-4740-a9e4-328091ee920f-istio-data" pod "9e6f829a-4773-4740-a9e4-328091ee920f" (UID: "9e6f829a-4773-4740-a9e4-328091ee920f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:48:14.841281 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.840785 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-istio-data" pod "0f4f1372-1a38-4d0b-b73b-9a9fa8b881de" (UID: "0f4f1372-1a38-4d0b-b73b-9a9fa8b881de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:48:14.841281 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.840985 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e6f829a-4773-4740-a9e4-328091ee920f-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:14.841281 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.841007 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6760e57-3c38-4dda-84e0-75b99b3a70a6-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:14.841281 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.841026 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtz6r\" (UniqueName: \"kubernetes.io/projected/b6760e57-3c38-4dda-84e0-75b99b3a70a6-kube-api-access-jtz6r\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:14.841281 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.841042 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6760e57-3c38-4dda-84e0-75b99b3a70a6-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:14.841281 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.841057 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0ce9454-b676-4e7d-abd3-34d76aa2c787-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:14.841281 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.841070 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0ce9454-b676-4e7d-abd3-34d76aa2c787-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:14.841281 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.841084 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:14.841281 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.841099 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln9vr\" (UniqueName: \"kubernetes.io/projected/b0ce9454-b676-4e7d-abd3-34d76aa2c787-kube-api-access-ln9vr\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:14.841281 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.841114 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6760e57-3c38-4dda-84e0-75b99b3a70a6-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:14.841281 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.841128 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e6f829a-4773-4740-a9e4-328091ee920f-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:14.841281 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.841144 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6760e57-3c38-4dda-84e0-75b99b3a70a6-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:14.841281 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.841151 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-istiod-ca-cert" pod "0f4f1372-1a38-4d0b-b73b-9a9fa8b881de" (UID: "0f4f1372-1a38-4d0b-b73b-9a9fa8b881de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:48:14.841281 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.841160 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e6f829a-4773-4740-a9e4-328091ee920f-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:14.841281 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.841199 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0ce9454-b676-4e7d-abd3-34d76aa2c787-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:14.841281 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.841217 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e6f829a-4773-4740-a9e4-328091ee920f-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:14.841281 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.841232 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:14.841281 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.841249 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:14.841281 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.841264 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:14.841281 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.841282 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e6f829a-4773-4740-a9e4-328091ee920f-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:14.842836 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.842798 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e6f829a-4773-4740-a9e4-328091ee920f-kube-api-access-tl257" pod "9e6f829a-4773-4740-a9e4-328091ee920f" (UID: "9e6f829a-4773-4740-a9e4-328091ee920f"). InnerVolumeSpecName "kube-api-access-tl257". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:48:14.842968 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.842859 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e6f829a-4773-4740-a9e4-328091ee920f-istio-token" pod "9e6f829a-4773-4740-a9e4-328091ee920f" (UID: "9e6f829a-4773-4740-a9e4-328091ee920f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:48:14.843046 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.842994 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-istio-token" pod "0f4f1372-1a38-4d0b-b73b-9a9fa8b881de" (UID: "0f4f1372-1a38-4d0b-b73b-9a9fa8b881de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:48:14.843230 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.843205 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-kube-api-access-6d9xk" pod "0f4f1372-1a38-4d0b-b73b-9a9fa8b881de" (UID: "0f4f1372-1a38-4d0b-b73b-9a9fa8b881de"). InnerVolumeSpecName "kube-api-access-6d9xk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:48:14.843355 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.843336 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e6f829a-4773-4740-a9e4-328091ee920f-istio-envoy" pod "9e6f829a-4773-4740-a9e4-328091ee920f" (UID: "9e6f829a-4773-4740-a9e4-328091ee920f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:48:14.843505 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.843470 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-istio-podinfo" pod "0f4f1372-1a38-4d0b-b73b-9a9fa8b881de" (UID: "0f4f1372-1a38-4d0b-b73b-9a9fa8b881de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:48:14.843581 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.843562 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-istio-envoy" pod "0f4f1372-1a38-4d0b-b73b-9a9fa8b881de" (UID: "0f4f1372-1a38-4d0b-b73b-9a9fa8b881de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:48:14.844032 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.844014 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e6f829a-4773-4740-a9e4-328091ee920f-istio-podinfo" pod "9e6f829a-4773-4740-a9e4-328091ee920f" (UID: "9e6f829a-4773-4740-a9e4-328091ee920f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:48:14.942704 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.942656 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl257\" (UniqueName: \"kubernetes.io/projected/9e6f829a-4773-4740-a9e4-328091ee920f-kube-api-access-tl257\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:14.942704 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.942703 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:14.943035 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.942717 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:14.943035 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.942729 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:14.943035 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.942742 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e6f829a-4773-4740-a9e4-328091ee920f-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:14.943035 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.942754 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e6f829a-4773-4740-a9e4-328091ee920f-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:14.943035 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.942767 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e6f829a-4773-4740-a9e4-328091ee920f-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:14.943035 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.942780 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:14.943035 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:14.942793 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d9xk\" (UniqueName: \"kubernetes.io/projected/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de-kube-api-access-6d9xk\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:15.420897 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:15.420860 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8"] Nov 01 06:48:15.425971 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:15.425937 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jxfb8"] Nov 01 06:48:15.515926 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:15.515893 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6760e57-3c38-4dda-84e0-75b99b3a70a6" path="/var/lib/kubelet/pods/b6760e57-3c38-4dda-84e0-75b99b3a70a6/volumes" Nov 01 06:48:15.533903 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:15.533874 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl"] Nov 01 06:48:15.538968 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:15.538944 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-brrdl"] Nov 01 06:48:15.683159 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:15.683070 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" Nov 01 06:48:15.683159 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:15.683085 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x" event={"ID":"0f4f1372-1a38-4d0b-b73b-9a9fa8b881de","Type":"ContainerDied","Data":"869ac66553d4d9269e9ce25d3a01bf560a07cfe4790d757b867905bcd43255f5"} Nov 01 06:48:15.683679 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:15.683183 5040 scope.go:122] "RemoveContainer" containerID="20708662581ab371ed8b56bb6d07bff07a8f8d2a5dcf7ef260ea6f26dfb987e3" Nov 01 06:48:15.685939 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:15.685919 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" Nov 01 06:48:15.686076 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:15.686014 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj" event={"ID":"9e6f829a-4773-4740-a9e4-328091ee920f","Type":"ContainerDied","Data":"4849ae5aa5919e938b3490a481560d993b0eafa430e441ea9434f24bf45d78d9"} Nov 01 06:48:15.693688 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:15.693669 5040 scope.go:122] "RemoveContainer" containerID="ae843ba1e8e079961f95d588c37ea16c76aeb714ead9448c841153ddbb0f57c0" Nov 01 06:48:15.840805 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:15.840779 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67d7f00-xmj9q" Nov 01 06:48:15.850678 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:15.850651 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj"] Nov 01 06:48:15.854565 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:15.854542 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pnbfj"] Nov 01 06:48:15.954671 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:15.954592 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb479226-2d4d-470f-9202-2ca474ad9f5c-kube-api-access-4kbb7\" (UniqueName: \"kubernetes.io/projected/fb479226-2d4d-470f-9202-2ca474ad9f5c-kube-api-access-4kbb7\") pod \"fb479226-2d4d-470f-9202-2ca474ad9f5c\" (UID: \"fb479226-2d4d-470f-9202-2ca474ad9f5c\") " Nov 01 06:48:15.954835 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:15.954703 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb479226-2d4d-470f-9202-2ca474ad9f5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb479226-2d4d-470f-9202-2ca474ad9f5c-maas-api-openshift-ca\") pod \"fb479226-2d4d-470f-9202-2ca474ad9f5c\" (UID: \"fb479226-2d4d-470f-9202-2ca474ad9f5c\") " Nov 01 06:48:15.954835 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:15.954754 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb479226-2d4d-470f-9202-2ca474ad9f5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb479226-2d4d-470f-9202-2ca474ad9f5c-maas-api-xks-ca\") pod \"fb479226-2d4d-470f-9202-2ca474ad9f5c\" (UID: \"fb479226-2d4d-470f-9202-2ca474ad9f5c\") " Nov 01 06:48:15.955065 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:15.955038 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb479226-2d4d-470f-9202-2ca474ad9f5c-maas-api-openshift-ca" pod "fb479226-2d4d-470f-9202-2ca474ad9f5c" (UID: "fb479226-2d4d-470f-9202-2ca474ad9f5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:48:15.956881 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:15.956848 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb479226-2d4d-470f-9202-2ca474ad9f5c-maas-api-xks-ca" pod "fb479226-2d4d-470f-9202-2ca474ad9f5c" (UID: "fb479226-2d4d-470f-9202-2ca474ad9f5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:48:15.956881 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:15.956866 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb479226-2d4d-470f-9202-2ca474ad9f5c-kube-api-access-4kbb7" pod "fb479226-2d4d-470f-9202-2ca474ad9f5c" (UID: "fb479226-2d4d-470f-9202-2ca474ad9f5c"). InnerVolumeSpecName "kube-api-access-4kbb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:48:15.965878 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:15.965850 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x"] Nov 01 06:48:15.969801 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:15.969776 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8th8x"] Nov 01 06:48:16.055671 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:16.055627 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb479226-2d4d-470f-9202-2ca474ad9f5c-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:16.055671 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:16.055664 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kbb7\" (UniqueName: \"kubernetes.io/projected/fb479226-2d4d-470f-9202-2ca474ad9f5c-kube-api-access-4kbb7\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:16.055671 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:16.055679 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb479226-2d4d-470f-9202-2ca474ad9f5c-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:16.695391 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:16.695358 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67d7f00-xmj9q" Nov 01 06:48:16.695933 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:16.695406 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67d7f00-xmj9q" event={"ID":"fb479226-2d4d-470f-9202-2ca474ad9f5c","Type":"ContainerDied","Data":"bc8fdc6a09e62a03e51e0ed60047a6a5d32b3bb186cf876a252bdb803c5fbf94"} Nov 01 06:48:16.695933 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:16.695520 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc8fdc6a09e62a03e51e0ed60047a6a5d32b3bb186cf876a252bdb803c5fbf94" Nov 01 06:48:17.517647 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:17.517602 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f4f1372-1a38-4d0b-b73b-9a9fa8b881de" path="/var/lib/kubelet/pods/0f4f1372-1a38-4d0b-b73b-9a9fa8b881de/volumes" Nov 01 06:48:17.518358 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:17.518335 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e6f829a-4773-4740-a9e4-328091ee920f" path="/var/lib/kubelet/pods/9e6f829a-4773-4740-a9e4-328091ee920f/volumes" Nov 01 06:48:17.519041 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:17.519020 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0ce9454-b676-4e7d-abd3-34d76aa2c787" path="/var/lib/kubelet/pods/b0ce9454-b676-4e7d-abd3-34d76aa2c787/volumes" Nov 01 06:48:22.811902 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:22.811868 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c03cdb9-zld5v"] Nov 01 06:48:22.812371 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:22.812356 5040 cpu_manager.go:414] "containerMap: removing container" podUID="9e6f829a-4773-4740-a9e4-328091ee920f" containerName="istio-proxy" Nov 01 06:48:22.812415 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:22.812372 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e6f829a-4773-4740-a9e4-328091ee920f" containerName="istio-proxy" Nov 01 06:48:22.812415 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:22.812390 5040 cpu_manager.go:414] "containerMap: removing container" podUID="b6760e57-3c38-4dda-84e0-75b99b3a70a6" containerName="istio-proxy" Nov 01 06:48:22.812415 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:22.812395 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6760e57-3c38-4dda-84e0-75b99b3a70a6" containerName="istio-proxy" Nov 01 06:48:22.812415 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:22.812403 5040 cpu_manager.go:414] "containerMap: removing container" podUID="0f4f1372-1a38-4d0b-b73b-9a9fa8b881de" containerName="istio-proxy" Nov 01 06:48:22.812415 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:22.812408 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f4f1372-1a38-4d0b-b73b-9a9fa8b881de" containerName="istio-proxy" Nov 01 06:48:22.812609 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:22.812431 5040 cpu_manager.go:414] "containerMap: removing container" podUID="b0ce9454-b676-4e7d-abd3-34d76aa2c787" containerName="istio-proxy" Nov 01 06:48:22.812609 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:22.812436 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ce9454-b676-4e7d-abd3-34d76aa2c787" containerName="istio-proxy" Nov 01 06:48:22.812609 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:22.812444 5040 cpu_manager.go:414] "containerMap: removing container" podUID="fb479226-2d4d-470f-9202-2ca474ad9f5c" containerName="revoke-keys" Nov 01 06:48:22.812609 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:22.812451 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb479226-2d4d-470f-9202-2ca474ad9f5c" containerName="revoke-keys" Nov 01 06:48:22.812609 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:22.812541 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ce9454-b676-4e7d-abd3-34d76aa2c787" containerName="istio-proxy" Nov 01 06:48:22.812609 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:22.812551 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb479226-2d4d-470f-9202-2ca474ad9f5c" containerName="revoke-keys" Nov 01 06:48:22.812609 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:22.812561 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e6f829a-4773-4740-a9e4-328091ee920f" containerName="istio-proxy" Nov 01 06:48:22.812609 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:22.812568 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6760e57-3c38-4dda-84e0-75b99b3a70a6" containerName="istio-proxy" Nov 01 06:48:22.812609 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:22.812574 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f4f1372-1a38-4d0b-b73b-9a9fa8b881de" containerName="istio-proxy" Nov 01 06:48:22.817139 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:22.817120 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c03cdb9-zld5v" Nov 01 06:48:22.832401 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:22.832377 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zmj6\" (UniqueName: \"kubernetes.io/projected/c623288c-7e7e-413f-9ee4-7b7bc14212d8-kube-api-access-5zmj6\") pod \"maas-api-revoke-sub-5c03cdb9-zld5v\" (UID: \"c623288c-7e7e-413f-9ee4-7b7bc14212d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c03cdb9-zld5v" Nov 01 06:48:22.832534 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:22.832443 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c623288c-7e7e-413f-9ee4-7b7bc14212d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c03cdb9-zld5v\" (UID: \"c623288c-7e7e-413f-9ee4-7b7bc14212d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c03cdb9-zld5v" Nov 01 06:48:22.832608 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:22.832586 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c623288c-7e7e-413f-9ee4-7b7bc14212d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c03cdb9-zld5v\" (UID: \"c623288c-7e7e-413f-9ee4-7b7bc14212d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c03cdb9-zld5v" Nov 01 06:48:22.870861 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:22.870830 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:48:22.871021 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:22.870929 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:48:22.922793 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:22.922754 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:48:22.922986 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:22.922821 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:48:22.926358 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:22.926324 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c03cdb9-zld5v"] Nov 01 06:48:22.933056 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:22.933033 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c623288c-7e7e-413f-9ee4-7b7bc14212d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c03cdb9-zld5v\" (UID: \"c623288c-7e7e-413f-9ee4-7b7bc14212d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c03cdb9-zld5v" Nov 01 06:48:22.933206 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:22.933185 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zmj6\" (UniqueName: \"kubernetes.io/projected/c623288c-7e7e-413f-9ee4-7b7bc14212d8-kube-api-access-5zmj6\") pod \"maas-api-revoke-sub-5c03cdb9-zld5v\" (UID: \"c623288c-7e7e-413f-9ee4-7b7bc14212d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c03cdb9-zld5v" Nov 01 06:48:22.933355 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:22.933337 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c623288c-7e7e-413f-9ee4-7b7bc14212d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c03cdb9-zld5v\" (UID: \"c623288c-7e7e-413f-9ee4-7b7bc14212d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c03cdb9-zld5v" Nov 01 06:48:22.934424 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:22.934394 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c623288c-7e7e-413f-9ee4-7b7bc14212d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c03cdb9-zld5v\" (UID: \"c623288c-7e7e-413f-9ee4-7b7bc14212d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c03cdb9-zld5v" Nov 01 06:48:22.935647 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:22.935627 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c623288c-7e7e-413f-9ee4-7b7bc14212d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c03cdb9-zld5v\" (UID: \"c623288c-7e7e-413f-9ee4-7b7bc14212d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c03cdb9-zld5v" Nov 01 06:48:22.994805 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:22.994774 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zmj6\" (UniqueName: \"kubernetes.io/projected/c623288c-7e7e-413f-9ee4-7b7bc14212d8-kube-api-access-5zmj6\") pod \"maas-api-revoke-sub-5c03cdb9-zld5v\" (UID: \"c623288c-7e7e-413f-9ee4-7b7bc14212d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c03cdb9-zld5v" Nov 01 06:48:23.130226 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:23.130142 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c03cdb9-zld5v" Nov 01 06:48:23.508926 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:48:23.508891 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc623288c_7e7e_413f_9ee4_7b7bc14212d8.slice/crio-2973b3f4cac6581a1a23bdd6ee9a77fe7d6bfc62f9d731515666c3cbedfbff0b WatchSource:0}: Error finding container 2973b3f4cac6581a1a23bdd6ee9a77fe7d6bfc62f9d731515666c3cbedfbff0b: Status 404 returned error can't find the container with id 2973b3f4cac6581a1a23bdd6ee9a77fe7d6bfc62f9d731515666c3cbedfbff0b Nov 01 06:48:23.517952 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:23.517921 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c03cdb9-zld5v"] Nov 01 06:48:23.747804 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:23.747761 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c03cdb9-zld5v" event={"ID":"c623288c-7e7e-413f-9ee4-7b7bc14212d8","Type":"ContainerStarted","Data":"2973b3f4cac6581a1a23bdd6ee9a77fe7d6bfc62f9d731515666c3cbedfbff0b"} Nov 01 06:48:23.845319 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:23.845286 5040 scope.go:122] "RemoveContainer" containerID="8a3e1ae9ca705270509db3fbcf635c70ff4e4e776d83e77beca7d342560dbb6b" Nov 01 06:48:23.854919 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:23.854894 5040 scope.go:122] "RemoveContainer" containerID="6cd0db785c6439ed65ef248006198d8caca0c489ffe5cc0eb616a12e1ca16aed" Nov 01 06:48:24.755749 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:24.755712 5040 generic.go:356] "Generic (PLEG): container finished" podID="c623288c-7e7e-413f-9ee4-7b7bc14212d8" containerID="20fa5ab762cb05b47614d8bbc6fc748cb352f13024679cf7570147924df1f700" exitCode=0 Nov 01 06:48:24.755939 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:24.755788 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c03cdb9-zld5v" event={"ID":"c623288c-7e7e-413f-9ee4-7b7bc14212d8","Type":"ContainerDied","Data":"20fa5ab762cb05b47614d8bbc6fc748cb352f13024679cf7570147924df1f700"} Nov 01 06:48:25.914336 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:25.914309 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c03cdb9-zld5v" Nov 01 06:48:25.964776 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:25.964738 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c623288c-7e7e-413f-9ee4-7b7bc14212d8-kube-api-access-5zmj6\" (UniqueName: \"kubernetes.io/projected/c623288c-7e7e-413f-9ee4-7b7bc14212d8-kube-api-access-5zmj6\") pod \"c623288c-7e7e-413f-9ee4-7b7bc14212d8\" (UID: \"c623288c-7e7e-413f-9ee4-7b7bc14212d8\") " Nov 01 06:48:25.964993 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:25.964845 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c623288c-7e7e-413f-9ee4-7b7bc14212d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c623288c-7e7e-413f-9ee4-7b7bc14212d8-maas-api-xks-ca\") pod \"c623288c-7e7e-413f-9ee4-7b7bc14212d8\" (UID: \"c623288c-7e7e-413f-9ee4-7b7bc14212d8\") " Nov 01 06:48:25.964993 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:25.964924 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c623288c-7e7e-413f-9ee4-7b7bc14212d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c623288c-7e7e-413f-9ee4-7b7bc14212d8-maas-api-openshift-ca\") pod \"c623288c-7e7e-413f-9ee4-7b7bc14212d8\" (UID: \"c623288c-7e7e-413f-9ee4-7b7bc14212d8\") " Nov 01 06:48:25.965376 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:25.965341 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c623288c-7e7e-413f-9ee4-7b7bc14212d8-maas-api-openshift-ca" pod "c623288c-7e7e-413f-9ee4-7b7bc14212d8" (UID: "c623288c-7e7e-413f-9ee4-7b7bc14212d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:48:25.968611 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:25.967361 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c623288c-7e7e-413f-9ee4-7b7bc14212d8-maas-api-xks-ca" pod "c623288c-7e7e-413f-9ee4-7b7bc14212d8" (UID: "c623288c-7e7e-413f-9ee4-7b7bc14212d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:48:25.969206 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:25.969057 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c623288c-7e7e-413f-9ee4-7b7bc14212d8-kube-api-access-5zmj6" pod "c623288c-7e7e-413f-9ee4-7b7bc14212d8" (UID: "c623288c-7e7e-413f-9ee4-7b7bc14212d8"). InnerVolumeSpecName "kube-api-access-5zmj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:48:26.066313 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:26.066223 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zmj6\" (UniqueName: \"kubernetes.io/projected/c623288c-7e7e-413f-9ee4-7b7bc14212d8-kube-api-access-5zmj6\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:26.066313 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:26.066260 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c623288c-7e7e-413f-9ee4-7b7bc14212d8-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:26.066313 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:26.066274 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c623288c-7e7e-413f-9ee4-7b7bc14212d8-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:26.770804 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:26.770765 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c03cdb9-zld5v" Nov 01 06:48:26.771022 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:26.770797 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c03cdb9-zld5v" event={"ID":"c623288c-7e7e-413f-9ee4-7b7bc14212d8","Type":"ContainerDied","Data":"2973b3f4cac6581a1a23bdd6ee9a77fe7d6bfc62f9d731515666c3cbedfbff0b"} Nov 01 06:48:26.771022 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:26.770856 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2973b3f4cac6581a1a23bdd6ee9a77fe7d6bfc62f9d731515666c3cbedfbff0b" Nov 01 06:48:32.047842 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:32.047802 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b3cdfa-z9swx"] Nov 01 06:48:32.098354 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:32.098316 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b3cdfa-z9swx"] Nov 01 06:48:33.515191 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:33.515150 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f" path="/var/lib/kubelet/pods/c6a37b7c-e9ef-4112-9a1f-ff8122c18a5f/volumes" Nov 01 06:48:39.973215 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:48:39.973174 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:48:53.048546 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:53.048503 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-925683cf-97wz6"] Nov 01 06:48:53.098204 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:53.098170 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-925683cf-97wz6"] Nov 01 06:48:53.515149 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:48:53.515113 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09e76dfd-4002-462f-85bd-38d81d2b4046" path="/var/lib/kubelet/pods/09e76dfd-4002-462f-85bd-38d81d2b4046/volumes" Nov 01 06:49:09.972248 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:49:09.972212 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:49:15.597198 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:15.597159 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dc4bced-jxvvg"] Nov 01 06:49:15.597861 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:15.597837 5040 cpu_manager.go:414] "containerMap: removing container" podUID="c623288c-7e7e-413f-9ee4-7b7bc14212d8" containerName="revoke-keys" Nov 01 06:49:15.597964 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:15.597862 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c623288c-7e7e-413f-9ee4-7b7bc14212d8" containerName="revoke-keys" Nov 01 06:49:15.598041 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:15.598025 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="c623288c-7e7e-413f-9ee4-7b7bc14212d8" containerName="revoke-keys" Nov 01 06:49:15.601532 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:15.601510 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc4bced-jxvvg" Nov 01 06:49:15.659791 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:15.659758 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dc4bced-jxvvg"] Nov 01 06:49:15.705780 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:15.705740 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:49:15.705991 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:15.705823 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:49:15.705991 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:15.705843 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:49:15.705991 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:15.705823 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:49:15.707114 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:15.707095 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aef17fca-e57d-40af-bc03-88322b9d508e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dc4bced-jxvvg\" (UID: \"aef17fca-e57d-40af-bc03-88322b9d508e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc4bced-jxvvg" Nov 01 06:49:15.707168 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:15.707153 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbx6l\" (UniqueName: \"kubernetes.io/projected/aef17fca-e57d-40af-bc03-88322b9d508e-kube-api-access-pbx6l\") pod \"maas-api-revoke-sub-1dc4bced-jxvvg\" (UID: \"aef17fca-e57d-40af-bc03-88322b9d508e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc4bced-jxvvg" Nov 01 06:49:15.707310 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:15.707288 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aef17fca-e57d-40af-bc03-88322b9d508e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dc4bced-jxvvg\" (UID: \"aef17fca-e57d-40af-bc03-88322b9d508e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc4bced-jxvvg" Nov 01 06:49:15.808033 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:15.808000 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aef17fca-e57d-40af-bc03-88322b9d508e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dc4bced-jxvvg\" (UID: \"aef17fca-e57d-40af-bc03-88322b9d508e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc4bced-jxvvg" Nov 01 06:49:15.808235 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:15.808075 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aef17fca-e57d-40af-bc03-88322b9d508e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dc4bced-jxvvg\" (UID: \"aef17fca-e57d-40af-bc03-88322b9d508e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc4bced-jxvvg" Nov 01 06:49:15.808235 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:15.808120 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbx6l\" (UniqueName: \"kubernetes.io/projected/aef17fca-e57d-40af-bc03-88322b9d508e-kube-api-access-pbx6l\") pod \"maas-api-revoke-sub-1dc4bced-jxvvg\" (UID: \"aef17fca-e57d-40af-bc03-88322b9d508e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc4bced-jxvvg" Nov 01 06:49:15.808853 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:15.808825 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aef17fca-e57d-40af-bc03-88322b9d508e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dc4bced-jxvvg\" (UID: \"aef17fca-e57d-40af-bc03-88322b9d508e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc4bced-jxvvg" Nov 01 06:49:15.810414 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:15.810390 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aef17fca-e57d-40af-bc03-88322b9d508e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dc4bced-jxvvg\" (UID: \"aef17fca-e57d-40af-bc03-88322b9d508e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc4bced-jxvvg" Nov 01 06:49:15.870070 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:15.869998 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbx6l\" (UniqueName: \"kubernetes.io/projected/aef17fca-e57d-40af-bc03-88322b9d508e-kube-api-access-pbx6l\") pod \"maas-api-revoke-sub-1dc4bced-jxvvg\" (UID: \"aef17fca-e57d-40af-bc03-88322b9d508e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc4bced-jxvvg" Nov 01 06:49:15.914573 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:15.914541 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc4bced-jxvvg" Nov 01 06:49:16.290399 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:16.290371 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dc4bced-jxvvg"] Nov 01 06:49:16.291407 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:49:16.291382 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaef17fca_e57d_40af_bc03_88322b9d508e.slice/crio-1c2c3312603fc423277013e7aab884866c268ba908dd41ff7cf82e4b34232e41 WatchSource:0}: Error finding container 1c2c3312603fc423277013e7aab884866c268ba908dd41ff7cf82e4b34232e41: Status 404 returned error can't find the container with id 1c2c3312603fc423277013e7aab884866c268ba908dd41ff7cf82e4b34232e41 Nov 01 06:49:17.050284 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:17.050246 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd40336b-gpg8j"] Nov 01 06:49:17.081994 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:17.081963 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd40336b-gpg8j"] Nov 01 06:49:17.117710 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:17.117671 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc4bced-jxvvg" event={"ID":"aef17fca-e57d-40af-bc03-88322b9d508e","Type":"ContainerStarted","Data":"2a7ab790ba66b2b8128aac6790e41ba2d2f668797886fda5e77137aa912fcfb0"} Nov 01 06:49:17.117915 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:17.117766 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc4bced-jxvvg" event={"ID":"aef17fca-e57d-40af-bc03-88322b9d508e","Type":"ContainerStarted","Data":"1c2c3312603fc423277013e7aab884866c268ba908dd41ff7cf82e4b34232e41"} Nov 01 06:49:17.235677 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:17.235626 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc4bced-jxvvg" podStartSLOduration=2.235611971 podStartE2EDuration="2.235611971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:49:17.233999578 +0000 UTC m=+3835.889116746" watchObservedRunningTime="2026-11-01 06:49:17.235611971 +0000 UTC m=+3835.890729183" Nov 01 06:49:17.517854 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:17.517818 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97719997-cc0f-45b0-9816-887a62dde1e4" path="/var/lib/kubelet/pods/97719997-cc0f-45b0-9816-887a62dde1e4/volumes" Nov 01 06:49:18.125846 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:18.125810 5040 generic.go:356] "Generic (PLEG): container finished" podID="aef17fca-e57d-40af-bc03-88322b9d508e" containerID="2a7ab790ba66b2b8128aac6790e41ba2d2f668797886fda5e77137aa912fcfb0" exitCode=0 Nov 01 06:49:18.126250 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:18.125893 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc4bced-jxvvg" event={"ID":"aef17fca-e57d-40af-bc03-88322b9d508e","Type":"ContainerDied","Data":"2a7ab790ba66b2b8128aac6790e41ba2d2f668797886fda5e77137aa912fcfb0"} Nov 01 06:49:19.275852 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:19.275824 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc4bced-jxvvg" Nov 01 06:49:19.340871 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:19.340839 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aef17fca-e57d-40af-bc03-88322b9d508e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aef17fca-e57d-40af-bc03-88322b9d508e-maas-api-openshift-ca\") pod \"aef17fca-e57d-40af-bc03-88322b9d508e\" (UID: \"aef17fca-e57d-40af-bc03-88322b9d508e\") " Nov 01 06:49:19.341016 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:19.340886 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aef17fca-e57d-40af-bc03-88322b9d508e-kube-api-access-pbx6l\" (UniqueName: \"kubernetes.io/projected/aef17fca-e57d-40af-bc03-88322b9d508e-kube-api-access-pbx6l\") pod \"aef17fca-e57d-40af-bc03-88322b9d508e\" (UID: \"aef17fca-e57d-40af-bc03-88322b9d508e\") " Nov 01 06:49:19.341016 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:19.340925 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aef17fca-e57d-40af-bc03-88322b9d508e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aef17fca-e57d-40af-bc03-88322b9d508e-maas-api-xks-ca\") pod \"aef17fca-e57d-40af-bc03-88322b9d508e\" (UID: \"aef17fca-e57d-40af-bc03-88322b9d508e\") " Nov 01 06:49:19.341189 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:19.341167 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aef17fca-e57d-40af-bc03-88322b9d508e-maas-api-openshift-ca" pod "aef17fca-e57d-40af-bc03-88322b9d508e" (UID: "aef17fca-e57d-40af-bc03-88322b9d508e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:49:19.341423 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:19.341400 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aef17fca-e57d-40af-bc03-88322b9d508e-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:19.343046 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:19.343017 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aef17fca-e57d-40af-bc03-88322b9d508e-maas-api-xks-ca" pod "aef17fca-e57d-40af-bc03-88322b9d508e" (UID: "aef17fca-e57d-40af-bc03-88322b9d508e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:49:19.343046 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:19.343038 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aef17fca-e57d-40af-bc03-88322b9d508e-kube-api-access-pbx6l" pod "aef17fca-e57d-40af-bc03-88322b9d508e" (UID: "aef17fca-e57d-40af-bc03-88322b9d508e"). InnerVolumeSpecName "kube-api-access-pbx6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:49:19.442607 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:19.442523 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbx6l\" (UniqueName: \"kubernetes.io/projected/aef17fca-e57d-40af-bc03-88322b9d508e-kube-api-access-pbx6l\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:19.442607 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:19.442562 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aef17fca-e57d-40af-bc03-88322b9d508e-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:19.888170 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:19.888130 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx"] Nov 01 06:49:19.892530 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:19.892501 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj"] Nov 01 06:49:19.940234 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:19.940198 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x"] Nov 01 06:49:19.944023 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:19.943996 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4"] Nov 01 06:49:20.141403 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:20.141317 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc4bced-jxvvg" Nov 01 06:49:20.141403 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:20.141346 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc4bced-jxvvg" event={"ID":"aef17fca-e57d-40af-bc03-88322b9d508e","Type":"ContainerDied","Data":"1c2c3312603fc423277013e7aab884866c268ba908dd41ff7cf82e4b34232e41"} Nov 01 06:49:20.141403 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:20.141390 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c2c3312603fc423277013e7aab884866c268ba908dd41ff7cf82e4b34232e41" Nov 01 06:49:23.906583 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:23.906540 5040 scope.go:122] "RemoveContainer" containerID="90cd631c2cfb34b2e931f4ec3922b7827e6a8abd0d935c8f51e3322589c03c17" Nov 01 06:49:23.917427 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:23.917400 5040 scope.go:122] "RemoveContainer" containerID="441f8987dda188003b200ad57f2623a4d5b7e472581a54162d67e4f63925843e" Nov 01 06:49:26.795348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:26.795310 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01a4e934-pwrh4"] Nov 01 06:49:26.795876 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:26.795818 5040 cpu_manager.go:414] "containerMap: removing container" podUID="aef17fca-e57d-40af-bc03-88322b9d508e" containerName="revoke-keys" Nov 01 06:49:26.795876 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:26.795832 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aef17fca-e57d-40af-bc03-88322b9d508e" containerName="revoke-keys" Nov 01 06:49:26.795967 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:26.795911 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="aef17fca-e57d-40af-bc03-88322b9d508e" containerName="revoke-keys" Nov 01 06:49:26.799105 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:26.799085 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a4e934-pwrh4" Nov 01 06:49:26.852592 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:26.852551 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:49:26.857113 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:26.857080 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01a4e934-pwrh4"] Nov 01 06:49:26.902886 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:26.902848 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:49:26.902886 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:26.902868 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:49:26.921458 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:26.921405 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c84pk\" (UniqueName: \"kubernetes.io/projected/ac0d876d-4d31-4b49-b5ed-98dce05ee7a0-kube-api-access-c84pk\") pod \"maas-api-revoke-sub-01a4e934-pwrh4\" (UID: \"ac0d876d-4d31-4b49-b5ed-98dce05ee7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a4e934-pwrh4" Nov 01 06:49:26.921672 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:26.921616 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac0d876d-4d31-4b49-b5ed-98dce05ee7a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01a4e934-pwrh4\" (UID: \"ac0d876d-4d31-4b49-b5ed-98dce05ee7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a4e934-pwrh4" Nov 01 06:49:26.921672 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:26.921663 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac0d876d-4d31-4b49-b5ed-98dce05ee7a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01a4e934-pwrh4\" (UID: \"ac0d876d-4d31-4b49-b5ed-98dce05ee7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a4e934-pwrh4" Nov 01 06:49:26.957097 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:26.957016 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:49:27.023221 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:27.023173 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c84pk\" (UniqueName: \"kubernetes.io/projected/ac0d876d-4d31-4b49-b5ed-98dce05ee7a0-kube-api-access-c84pk\") pod \"maas-api-revoke-sub-01a4e934-pwrh4\" (UID: \"ac0d876d-4d31-4b49-b5ed-98dce05ee7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a4e934-pwrh4" Nov 01 06:49:27.023450 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:27.023291 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac0d876d-4d31-4b49-b5ed-98dce05ee7a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01a4e934-pwrh4\" (UID: \"ac0d876d-4d31-4b49-b5ed-98dce05ee7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a4e934-pwrh4" Nov 01 06:49:27.023450 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:27.023331 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac0d876d-4d31-4b49-b5ed-98dce05ee7a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01a4e934-pwrh4\" (UID: \"ac0d876d-4d31-4b49-b5ed-98dce05ee7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a4e934-pwrh4" Nov 01 06:49:27.024093 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:27.024066 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac0d876d-4d31-4b49-b5ed-98dce05ee7a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01a4e934-pwrh4\" (UID: \"ac0d876d-4d31-4b49-b5ed-98dce05ee7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a4e934-pwrh4" Nov 01 06:49:27.025793 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:27.025774 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac0d876d-4d31-4b49-b5ed-98dce05ee7a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01a4e934-pwrh4\" (UID: \"ac0d876d-4d31-4b49-b5ed-98dce05ee7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a4e934-pwrh4" Nov 01 06:49:27.087536 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:27.087441 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c84pk\" (UniqueName: \"kubernetes.io/projected/ac0d876d-4d31-4b49-b5ed-98dce05ee7a0-kube-api-access-c84pk\") pod \"maas-api-revoke-sub-01a4e934-pwrh4\" (UID: \"ac0d876d-4d31-4b49-b5ed-98dce05ee7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a4e934-pwrh4" Nov 01 06:49:27.112015 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:27.111985 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a4e934-pwrh4" Nov 01 06:49:27.496756 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:27.496723 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01a4e934-pwrh4"] Nov 01 06:49:27.497245 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:49:27.497212 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac0d876d_4d31_4b49_b5ed_98dce05ee7a0.slice/crio-ea9cda5b59f21e29ef3871060bbd4919441163146d14962bc2c3c5fba4eb9193 WatchSource:0}: Error finding container ea9cda5b59f21e29ef3871060bbd4919441163146d14962bc2c3c5fba4eb9193: Status 404 returned error can't find the container with id ea9cda5b59f21e29ef3871060bbd4919441163146d14962bc2c3c5fba4eb9193 Nov 01 06:49:28.202062 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:28.201972 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a4e934-pwrh4" event={"ID":"ac0d876d-4d31-4b49-b5ed-98dce05ee7a0","Type":"ContainerStarted","Data":"428e8d09f6b4887f7222e4d534df73a9327106879f049cb76676ccf9f64233f9"} Nov 01 06:49:28.202062 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:28.202028 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a4e934-pwrh4" event={"ID":"ac0d876d-4d31-4b49-b5ed-98dce05ee7a0","Type":"ContainerStarted","Data":"ea9cda5b59f21e29ef3871060bbd4919441163146d14962bc2c3c5fba4eb9193"} Nov 01 06:49:28.323787 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:28.323727 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a4e934-pwrh4" podStartSLOduration=2.3237127060000002 podStartE2EDuration="2.323712706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:49:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:49:28.320719556 +0000 UTC m=+3846.975836723" watchObservedRunningTime="2026-11-01 06:49:28.323712706 +0000 UTC m=+3846.978829872" Nov 01 06:49:29.209742 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:29.209709 5040 generic.go:356] "Generic (PLEG): container finished" podID="ac0d876d-4d31-4b49-b5ed-98dce05ee7a0" containerID="428e8d09f6b4887f7222e4d534df73a9327106879f049cb76676ccf9f64233f9" exitCode=0 Nov 01 06:49:29.210147 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:29.209798 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a4e934-pwrh4" event={"ID":"ac0d876d-4d31-4b49-b5ed-98dce05ee7a0","Type":"ContainerDied","Data":"428e8d09f6b4887f7222e4d534df73a9327106879f049cb76676ccf9f64233f9"} Nov 01 06:49:30.365406 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:30.365382 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a4e934-pwrh4" Nov 01 06:49:30.461575 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:30.461538 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac0d876d-4d31-4b49-b5ed-98dce05ee7a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac0d876d-4d31-4b49-b5ed-98dce05ee7a0-maas-api-openshift-ca\") pod \"ac0d876d-4d31-4b49-b5ed-98dce05ee7a0\" (UID: \"ac0d876d-4d31-4b49-b5ed-98dce05ee7a0\") " Nov 01 06:49:30.461772 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:30.461663 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac0d876d-4d31-4b49-b5ed-98dce05ee7a0-kube-api-access-c84pk\" (UniqueName: \"kubernetes.io/projected/ac0d876d-4d31-4b49-b5ed-98dce05ee7a0-kube-api-access-c84pk\") pod \"ac0d876d-4d31-4b49-b5ed-98dce05ee7a0\" (UID: \"ac0d876d-4d31-4b49-b5ed-98dce05ee7a0\") " Nov 01 06:49:30.461772 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:30.461749 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac0d876d-4d31-4b49-b5ed-98dce05ee7a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac0d876d-4d31-4b49-b5ed-98dce05ee7a0-maas-api-xks-ca\") pod \"ac0d876d-4d31-4b49-b5ed-98dce05ee7a0\" (UID: \"ac0d876d-4d31-4b49-b5ed-98dce05ee7a0\") " Nov 01 06:49:30.461900 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:30.461876 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac0d876d-4d31-4b49-b5ed-98dce05ee7a0-maas-api-openshift-ca" pod "ac0d876d-4d31-4b49-b5ed-98dce05ee7a0" (UID: "ac0d876d-4d31-4b49-b5ed-98dce05ee7a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:49:30.462101 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:30.462084 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac0d876d-4d31-4b49-b5ed-98dce05ee7a0-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:30.463779 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:30.463758 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac0d876d-4d31-4b49-b5ed-98dce05ee7a0-kube-api-access-c84pk" pod "ac0d876d-4d31-4b49-b5ed-98dce05ee7a0" (UID: "ac0d876d-4d31-4b49-b5ed-98dce05ee7a0"). InnerVolumeSpecName "kube-api-access-c84pk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:49:30.463876 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:30.463806 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac0d876d-4d31-4b49-b5ed-98dce05ee7a0-maas-api-xks-ca" pod "ac0d876d-4d31-4b49-b5ed-98dce05ee7a0" (UID: "ac0d876d-4d31-4b49-b5ed-98dce05ee7a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:49:30.563764 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:30.563722 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c84pk\" (UniqueName: \"kubernetes.io/projected/ac0d876d-4d31-4b49-b5ed-98dce05ee7a0-kube-api-access-c84pk\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:30.563764 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:30.563761 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac0d876d-4d31-4b49-b5ed-98dce05ee7a0-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:31.046981 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:31.046948 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f00072c-vrlxg"] Nov 01 06:49:31.099671 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:31.099631 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f00072c-vrlxg"] Nov 01 06:49:31.223310 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:31.223272 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a4e934-pwrh4" Nov 01 06:49:31.223510 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:31.223308 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a4e934-pwrh4" event={"ID":"ac0d876d-4d31-4b49-b5ed-98dce05ee7a0","Type":"ContainerDied","Data":"ea9cda5b59f21e29ef3871060bbd4919441163146d14962bc2c3c5fba4eb9193"} Nov 01 06:49:31.223510 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:31.223368 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea9cda5b59f21e29ef3871060bbd4919441163146d14962bc2c3c5fba4eb9193" Nov 01 06:49:31.515978 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:31.515945 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b720868-e423-4738-bb3b-80f505d1bf2b" path="/var/lib/kubelet/pods/0b720868-e423-4738-bb3b-80f505d1bf2b/volumes" Nov 01 06:49:34.908976 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:34.908912 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" podUID="adbac784-4ded-4ca3-b985-3e59983fc637" containerName="main" containerID="cri-o://9a07bd563dd1c68552092f7bc868972a478ddabaeaf86c5d1c83bf7e4f262195" gracePeriod=45 Nov 01 06:49:34.912173 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:34.912109 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" podUID="75afe340-0b9c-4b4c-89ab-20d761e6b4f1" containerName="main" containerID="cri-o://dd00248788b40a58b0244afaa32709485526feaf81eb0fc7f6090c0bcaed951f" gracePeriod=45 Nov 01 06:49:34.962547 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:34.962510 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" podUID="57cf5e07-36de-48cf-b938-138484cea3a6" containerName="main" containerID="cri-o://7daac90a2fc5ccdcea8d31696984b7a7183171a108776bd09e43561ba3977eaa" gracePeriod=45 Nov 01 06:49:34.969816 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:34.969766 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" podUID="1878b6fb-f8fe-4f4f-8f4e-056c66fc206a" containerName="main" containerID="cri-o://d0dae898e24f105768cd27fbef7c830ee1cda3094e8cfd3b99fae27478517cf0" gracePeriod=45 Nov 01 06:49:35.259379 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.258977 5040 generic.go:356] "Generic (PLEG): container finished" podID="1878b6fb-f8fe-4f4f-8f4e-056c66fc206a" containerID="d0dae898e24f105768cd27fbef7c830ee1cda3094e8cfd3b99fae27478517cf0" exitCode=0 Nov 01 06:49:35.259379 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.259109 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" event={"ID":"1878b6fb-f8fe-4f4f-8f4e-056c66fc206a","Type":"ContainerDied","Data":"d0dae898e24f105768cd27fbef7c830ee1cda3094e8cfd3b99fae27478517cf0"} Nov 01 06:49:35.260934 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.260638 5040 generic.go:356] "Generic (PLEG): container finished" podID="75afe340-0b9c-4b4c-89ab-20d761e6b4f1" containerID="dd00248788b40a58b0244afaa32709485526feaf81eb0fc7f6090c0bcaed951f" exitCode=0 Nov 01 06:49:35.260934 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.260708 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" event={"ID":"75afe340-0b9c-4b4c-89ab-20d761e6b4f1","Type":"ContainerDied","Data":"dd00248788b40a58b0244afaa32709485526feaf81eb0fc7f6090c0bcaed951f"} Nov 01 06:49:35.260934 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.260741 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" event={"ID":"75afe340-0b9c-4b4c-89ab-20d761e6b4f1","Type":"ContainerDied","Data":"86d294025e93b037ed9348a4baf29761a468f90e797913038e1450aeba86f07e"} Nov 01 06:49:35.260934 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.260757 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86d294025e93b037ed9348a4baf29761a468f90e797913038e1450aeba86f07e" Nov 01 06:49:35.262527 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.262365 5040 generic.go:356] "Generic (PLEG): container finished" podID="57cf5e07-36de-48cf-b938-138484cea3a6" containerID="7daac90a2fc5ccdcea8d31696984b7a7183171a108776bd09e43561ba3977eaa" exitCode=0 Nov 01 06:49:35.262527 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.262441 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" event={"ID":"57cf5e07-36de-48cf-b938-138484cea3a6","Type":"ContainerDied","Data":"7daac90a2fc5ccdcea8d31696984b7a7183171a108776bd09e43561ba3977eaa"} Nov 01 06:49:35.264218 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.264141 5040 generic.go:356] "Generic (PLEG): container finished" podID="adbac784-4ded-4ca3-b985-3e59983fc637" containerID="9a07bd563dd1c68552092f7bc868972a478ddabaeaf86c5d1c83bf7e4f262195" exitCode=0 Nov 01 06:49:35.264218 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.264189 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" event={"ID":"adbac784-4ded-4ca3-b985-3e59983fc637","Type":"ContainerDied","Data":"9a07bd563dd1c68552092f7bc868972a478ddabaeaf86c5d1c83bf7e4f262195"} Nov 01 06:49:35.272034 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.272015 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" Nov 01 06:49:35.275033 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.275010 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" Nov 01 06:49:35.309547 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.309519 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" Nov 01 06:49:35.319724 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.319701 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" Nov 01 06:49:35.332453 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.332426 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57cf5e07-36de-48cf-b938-138484cea3a6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57cf5e07-36de-48cf-b938-138484cea3a6-model-cache\") pod \"57cf5e07-36de-48cf-b938-138484cea3a6\" (UID: \"57cf5e07-36de-48cf-b938-138484cea3a6\") " Nov 01 06:49:35.332614 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.332464 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57cf5e07-36de-48cf-b938-138484cea3a6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cf5e07-36de-48cf-b938-138484cea3a6-tmp-dir\") pod \"57cf5e07-36de-48cf-b938-138484cea3a6\" (UID: \"57cf5e07-36de-48cf-b938-138484cea3a6\") " Nov 01 06:49:35.332614 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.332521 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-dshm\") pod \"1878b6fb-f8fe-4f4f-8f4e-056c66fc206a\" (UID: \"1878b6fb-f8fe-4f4f-8f4e-056c66fc206a\") " Nov 01 06:49:35.332614 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.332552 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-tls-certs\" (UniqueName: \"kubernetes.io/secret/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-tls-certs\") pod \"75afe340-0b9c-4b4c-89ab-20d761e6b4f1\" (UID: \"75afe340-0b9c-4b4c-89ab-20d761e6b4f1\") " Nov 01 06:49:35.332614 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.332595 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-model-cache\") pod \"75afe340-0b9c-4b4c-89ab-20d761e6b4f1\" (UID: \"75afe340-0b9c-4b4c-89ab-20d761e6b4f1\") " Nov 01 06:49:35.332844 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.332623 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57cf5e07-36de-48cf-b938-138484cea3a6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/57cf5e07-36de-48cf-b938-138484cea3a6-dshm\") pod \"57cf5e07-36de-48cf-b938-138484cea3a6\" (UID: \"57cf5e07-36de-48cf-b938-138484cea3a6\") " Nov 01 06:49:35.332844 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.332683 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-home\" (UniqueName: \"kubernetes.io/empty-dir/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-home\") pod \"75afe340-0b9c-4b4c-89ab-20d761e6b4f1\" (UID: \"75afe340-0b9c-4b4c-89ab-20d761e6b4f1\") " Nov 01 06:49:35.332844 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.332714 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adbac784-4ded-4ca3-b985-3e59983fc637-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adbac784-4ded-4ca3-b985-3e59983fc637-tmp-dir\") pod \"adbac784-4ded-4ca3-b985-3e59983fc637\" (UID: \"adbac784-4ded-4ca3-b985-3e59983fc637\") " Nov 01 06:49:35.332844 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.332720 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cf5e07-36de-48cf-b938-138484cea3a6-model-cache" pod "57cf5e07-36de-48cf-b938-138484cea3a6" (UID: "57cf5e07-36de-48cf-b938-138484cea3a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:49:35.332844 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.332734 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cf5e07-36de-48cf-b938-138484cea3a6-tmp-dir" pod "57cf5e07-36de-48cf-b938-138484cea3a6" (UID: "57cf5e07-36de-48cf-b938-138484cea3a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:49:35.332844 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.332769 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-tmp-dir\") pod \"75afe340-0b9c-4b4c-89ab-20d761e6b4f1\" (UID: \"75afe340-0b9c-4b4c-89ab-20d761e6b4f1\") " Nov 01 06:49:35.332844 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.332814 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adbac784-4ded-4ca3-b985-3e59983fc637-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adbac784-4ded-4ca3-b985-3e59983fc637-model-cache\") pod \"adbac784-4ded-4ca3-b985-3e59983fc637\" (UID: \"adbac784-4ded-4ca3-b985-3e59983fc637\") " Nov 01 06:49:35.333219 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.332849 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-kube-api-access-chmgt\" (UniqueName: \"kubernetes.io/projected/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-kube-api-access-chmgt\") pod \"75afe340-0b9c-4b4c-89ab-20d761e6b4f1\" (UID: \"75afe340-0b9c-4b4c-89ab-20d761e6b4f1\") " Nov 01 06:49:35.333219 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.332879 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adbac784-4ded-4ca3-b985-3e59983fc637-kube-api-access-6j9c8\" (UniqueName: \"kubernetes.io/projected/adbac784-4ded-4ca3-b985-3e59983fc637-kube-api-access-6j9c8\") pod \"adbac784-4ded-4ca3-b985-3e59983fc637\" (UID: \"adbac784-4ded-4ca3-b985-3e59983fc637\") " Nov 01 06:49:35.333219 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.332908 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-model-cache" pod "75afe340-0b9c-4b4c-89ab-20d761e6b4f1" (UID: "75afe340-0b9c-4b4c-89ab-20d761e6b4f1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:49:35.333219 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.332927 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-home" pod "75afe340-0b9c-4b4c-89ab-20d761e6b4f1" (UID: "75afe340-0b9c-4b4c-89ab-20d761e6b4f1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:49:35.333219 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.332920 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-model-cache\") pod \"1878b6fb-f8fe-4f4f-8f4e-056c66fc206a\" (UID: \"1878b6fb-f8fe-4f4f-8f4e-056c66fc206a\") " Nov 01 06:49:35.333219 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.333014 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbac784-4ded-4ca3-b985-3e59983fc637-tmp-dir" pod "adbac784-4ded-4ca3-b985-3e59983fc637" (UID: "adbac784-4ded-4ca3-b985-3e59983fc637"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:49:35.333219 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.333038 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adbac784-4ded-4ca3-b985-3e59983fc637-tls-certs\" (UniqueName: \"kubernetes.io/secret/adbac784-4ded-4ca3-b985-3e59983fc637-tls-certs\") pod \"adbac784-4ded-4ca3-b985-3e59983fc637\" (UID: \"adbac784-4ded-4ca3-b985-3e59983fc637\") " Nov 01 06:49:35.333219 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.333080 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57cf5e07-36de-48cf-b938-138484cea3a6-kube-api-access-wrsfs\" (UniqueName: \"kubernetes.io/projected/57cf5e07-36de-48cf-b938-138484cea3a6-kube-api-access-wrsfs\") pod \"57cf5e07-36de-48cf-b938-138484cea3a6\" (UID: \"57cf5e07-36de-48cf-b938-138484cea3a6\") " Nov 01 06:49:35.333219 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.333119 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adbac784-4ded-4ca3-b985-3e59983fc637-home\" (UniqueName: \"kubernetes.io/empty-dir/adbac784-4ded-4ca3-b985-3e59983fc637-home\") pod \"adbac784-4ded-4ca3-b985-3e59983fc637\" (UID: \"adbac784-4ded-4ca3-b985-3e59983fc637\") " Nov 01 06:49:35.333219 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.333132 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-model-cache" pod "1878b6fb-f8fe-4f4f-8f4e-056c66fc206a" (UID: "1878b6fb-f8fe-4f4f-8f4e-056c66fc206a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:49:35.333219 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.333143 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbac784-4ded-4ca3-b985-3e59983fc637-model-cache" pod "adbac784-4ded-4ca3-b985-3e59983fc637" (UID: "adbac784-4ded-4ca3-b985-3e59983fc637"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:49:35.333219 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.333151 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-tls-certs\" (UniqueName: \"kubernetes.io/secret/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-tls-certs\") pod \"1878b6fb-f8fe-4f4f-8f4e-056c66fc206a\" (UID: \"1878b6fb-f8fe-4f4f-8f4e-056c66fc206a\") " Nov 01 06:49:35.333219 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.333202 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-dshm\") pod \"75afe340-0b9c-4b4c-89ab-20d761e6b4f1\" (UID: \"75afe340-0b9c-4b4c-89ab-20d761e6b4f1\") " Nov 01 06:49:35.333970 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.333232 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-kube-api-access-95jjb\" (UniqueName: \"kubernetes.io/projected/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-kube-api-access-95jjb\") pod \"1878b6fb-f8fe-4f4f-8f4e-056c66fc206a\" (UID: \"1878b6fb-f8fe-4f4f-8f4e-056c66fc206a\") " Nov 01 06:49:35.333970 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.333311 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adbac784-4ded-4ca3-b985-3e59983fc637-dshm\" (UniqueName: \"kubernetes.io/empty-dir/adbac784-4ded-4ca3-b985-3e59983fc637-dshm\") pod \"adbac784-4ded-4ca3-b985-3e59983fc637\" (UID: \"adbac784-4ded-4ca3-b985-3e59983fc637\") " Nov 01 06:49:35.333970 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.333345 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57cf5e07-36de-48cf-b938-138484cea3a6-tls-certs\" (UniqueName: \"kubernetes.io/secret/57cf5e07-36de-48cf-b938-138484cea3a6-tls-certs\") pod \"57cf5e07-36de-48cf-b938-138484cea3a6\" (UID: \"57cf5e07-36de-48cf-b938-138484cea3a6\") " Nov 01 06:49:35.333970 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.333364 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-tmp-dir" pod "75afe340-0b9c-4b4c-89ab-20d761e6b4f1" (UID: "75afe340-0b9c-4b4c-89ab-20d761e6b4f1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:49:35.333970 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.333389 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-tmp-dir\") pod \"1878b6fb-f8fe-4f4f-8f4e-056c66fc206a\" (UID: \"1878b6fb-f8fe-4f4f-8f4e-056c66fc206a\") " Nov 01 06:49:35.333970 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.333453 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-home\" (UniqueName: \"kubernetes.io/empty-dir/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-home\") pod \"1878b6fb-f8fe-4f4f-8f4e-056c66fc206a\" (UID: \"1878b6fb-f8fe-4f4f-8f4e-056c66fc206a\") " Nov 01 06:49:35.333970 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.333498 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57cf5e07-36de-48cf-b938-138484cea3a6-home\" (UniqueName: \"kubernetes.io/empty-dir/57cf5e07-36de-48cf-b938-138484cea3a6-home\") pod \"57cf5e07-36de-48cf-b938-138484cea3a6\" (UID: \"57cf5e07-36de-48cf-b938-138484cea3a6\") " Nov 01 06:49:35.333970 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.333926 5040 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57cf5e07-36de-48cf-b938-138484cea3a6-model-cache\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:35.333970 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.333948 5040 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cf5e07-36de-48cf-b938-138484cea3a6-tmp-dir\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:35.333970 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.333964 5040 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-model-cache\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:35.333970 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.333977 5040 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-home\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:35.335311 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.333993 5040 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adbac784-4ded-4ca3-b985-3e59983fc637-tmp-dir\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:35.335311 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.334008 5040 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-tmp-dir\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:35.335311 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.334020 5040 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adbac784-4ded-4ca3-b985-3e59983fc637-model-cache\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:35.335311 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.334033 5040 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-model-cache\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:35.335311 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.334229 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cf5e07-36de-48cf-b938-138484cea3a6-home" pod "57cf5e07-36de-48cf-b938-138484cea3a6" (UID: "57cf5e07-36de-48cf-b938-138484cea3a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:49:35.335311 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.334439 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-home" pod "1878b6fb-f8fe-4f4f-8f4e-056c66fc206a" (UID: "1878b6fb-f8fe-4f4f-8f4e-056c66fc206a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:49:35.335311 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.334431 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-tmp-dir" pod "1878b6fb-f8fe-4f4f-8f4e-056c66fc206a" (UID: "1878b6fb-f8fe-4f4f-8f4e-056c66fc206a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:49:35.335785 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.335625 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbac784-4ded-4ca3-b985-3e59983fc637-home" pod "adbac784-4ded-4ca3-b985-3e59983fc637" (UID: "adbac784-4ded-4ca3-b985-3e59983fc637"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:49:35.335785 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.335715 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adbac784-4ded-4ca3-b985-3e59983fc637-kube-api-access-6j9c8" pod "adbac784-4ded-4ca3-b985-3e59983fc637" (UID: "adbac784-4ded-4ca3-b985-3e59983fc637"). InnerVolumeSpecName "kube-api-access-6j9c8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:49:35.335945 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.335922 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-kube-api-access-chmgt" pod "75afe340-0b9c-4b4c-89ab-20d761e6b4f1" (UID: "75afe340-0b9c-4b4c-89ab-20d761e6b4f1"). InnerVolumeSpecName "kube-api-access-chmgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:49:35.337045 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.337019 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57cf5e07-36de-48cf-b938-138484cea3a6-tls-certs" pod "57cf5e07-36de-48cf-b938-138484cea3a6" (UID: "57cf5e07-36de-48cf-b938-138484cea3a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:49:35.337582 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.337459 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adbac784-4ded-4ca3-b985-3e59983fc637-tls-certs" pod "adbac784-4ded-4ca3-b985-3e59983fc637" (UID: "adbac784-4ded-4ca3-b985-3e59983fc637"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:49:35.338262 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.338233 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57cf5e07-36de-48cf-b938-138484cea3a6-kube-api-access-wrsfs" pod "57cf5e07-36de-48cf-b938-138484cea3a6" (UID: "57cf5e07-36de-48cf-b938-138484cea3a6"). InnerVolumeSpecName "kube-api-access-wrsfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:49:35.338368 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.338257 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-kube-api-access-95jjb" pod "1878b6fb-f8fe-4f4f-8f4e-056c66fc206a" (UID: "1878b6fb-f8fe-4f4f-8f4e-056c66fc206a"). InnerVolumeSpecName "kube-api-access-95jjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:49:35.338592 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.338569 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbac784-4ded-4ca3-b985-3e59983fc637-dshm" pod "adbac784-4ded-4ca3-b985-3e59983fc637" (UID: "adbac784-4ded-4ca3-b985-3e59983fc637"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:49:35.338935 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.338902 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-tls-certs" pod "1878b6fb-f8fe-4f4f-8f4e-056c66fc206a" (UID: "1878b6fb-f8fe-4f4f-8f4e-056c66fc206a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:49:35.339048 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.338983 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-dshm" pod "75afe340-0b9c-4b4c-89ab-20d761e6b4f1" (UID: "75afe340-0b9c-4b4c-89ab-20d761e6b4f1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:49:35.339166 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.339141 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-tls-certs" pod "75afe340-0b9c-4b4c-89ab-20d761e6b4f1" (UID: "75afe340-0b9c-4b4c-89ab-20d761e6b4f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:49:35.339280 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.339182 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-dshm" pod "1878b6fb-f8fe-4f4f-8f4e-056c66fc206a" (UID: "1878b6fb-f8fe-4f4f-8f4e-056c66fc206a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:49:35.340428 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.340398 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cf5e07-36de-48cf-b938-138484cea3a6-dshm" pod "57cf5e07-36de-48cf-b938-138484cea3a6" (UID: "57cf5e07-36de-48cf-b938-138484cea3a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:49:35.434848 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.434765 5040 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57cf5e07-36de-48cf-b938-138484cea3a6-dshm\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:35.434848 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.434798 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chmgt\" (UniqueName: \"kubernetes.io/projected/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-kube-api-access-chmgt\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:35.434848 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.434809 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j9c8\" (UniqueName: \"kubernetes.io/projected/adbac784-4ded-4ca3-b985-3e59983fc637-kube-api-access-6j9c8\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:35.434848 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.434819 5040 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adbac784-4ded-4ca3-b985-3e59983fc637-tls-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:35.434848 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.434829 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrsfs\" (UniqueName: \"kubernetes.io/projected/57cf5e07-36de-48cf-b938-138484cea3a6-kube-api-access-wrsfs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:35.434848 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.434837 5040 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adbac784-4ded-4ca3-b985-3e59983fc637-home\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:35.434848 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.434845 5040 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-tls-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:35.434848 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.434853 5040 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-dshm\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:35.435227 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.434862 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95jjb\" (UniqueName: \"kubernetes.io/projected/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-kube-api-access-95jjb\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:35.435227 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.434870 5040 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adbac784-4ded-4ca3-b985-3e59983fc637-dshm\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:35.435227 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.434878 5040 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57cf5e07-36de-48cf-b938-138484cea3a6-tls-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:35.435227 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.434886 5040 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-tmp-dir\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:35.435227 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.434894 5040 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-home\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:35.435227 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.434901 5040 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57cf5e07-36de-48cf-b938-138484cea3a6-home\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:35.435227 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.434908 5040 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a-dshm\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:35.435227 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:35.434915 5040 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75afe340-0b9c-4b4c-89ab-20d761e6b4f1-tls-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:36.272139 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:36.272105 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" Nov 01 06:49:36.272597 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:36.272136 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x" event={"ID":"57cf5e07-36de-48cf-b938-138484cea3a6","Type":"ContainerDied","Data":"c5021fe7b3298ee7e4717ac0e47c205fcacc86e0ed884b8fbba67982924ec859"} Nov 01 06:49:36.272597 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:36.272202 5040 scope.go:122] "RemoveContainer" containerID="7daac90a2fc5ccdcea8d31696984b7a7183171a108776bd09e43561ba3977eaa" Nov 01 06:49:36.273762 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:36.273743 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" Nov 01 06:49:36.273860 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:36.273785 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx" event={"ID":"adbac784-4ded-4ca3-b985-3e59983fc637","Type":"ContainerDied","Data":"a94537997b8356bb44c30a3838da4e7732486ca106d4d7fa3b05fe3d95e8a70e"} Nov 01 06:49:36.275312 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:36.275287 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" Nov 01 06:49:36.275312 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:36.275306 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj" Nov 01 06:49:36.275474 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:36.275329 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4" event={"ID":"1878b6fb-f8fe-4f4f-8f4e-056c66fc206a","Type":"ContainerDied","Data":"4a84add56d4c1d11c1360f4b96bbaab655138f1cc064e672cf5b54eaf8fd0ccd"} Nov 01 06:49:36.283593 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:36.283578 5040 scope.go:122] "RemoveContainer" containerID="9a07bd563dd1c68552092f7bc868972a478ddabaeaf86c5d1c83bf7e4f262195" Nov 01 06:49:36.292845 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:36.292824 5040 scope.go:122] "RemoveContainer" containerID="d0dae898e24f105768cd27fbef7c830ee1cda3094e8cfd3b99fae27478517cf0" Nov 01 06:49:36.440061 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:36.440024 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x"] Nov 01 06:49:36.445334 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:36.445306 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-14a6e4/premium-w2-14a6e4-kserve-57f7f854cc-l5v9x"] Nov 01 06:49:36.555561 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:36.555461 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj"] Nov 01 06:49:36.560382 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:36.560352 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-14a6e4/distinct-w2-14a6e4-kserve-85b55cc944-7wbnj"] Nov 01 06:49:36.671802 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:36.671768 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx"] Nov 01 06:49:36.675502 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:36.675455 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-14a6e4/distinct-2-w2-14a6e4-kserve-68d78d6c5-t4lgx"] Nov 01 06:49:36.789270 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:36.789233 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4"] Nov 01 06:49:36.797445 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:36.797410 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-14a6e4/simulator-w2-14a6e4-kserve-7b5c9758d4-rcjg4"] Nov 01 06:49:37.514460 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:37.514427 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1878b6fb-f8fe-4f4f-8f4e-056c66fc206a" path="/var/lib/kubelet/pods/1878b6fb-f8fe-4f4f-8f4e-056c66fc206a/volumes" Nov 01 06:49:37.514890 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:37.514875 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57cf5e07-36de-48cf-b938-138484cea3a6" path="/var/lib/kubelet/pods/57cf5e07-36de-48cf-b938-138484cea3a6/volumes" Nov 01 06:49:37.515255 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:37.515243 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75afe340-0b9c-4b4c-89ab-20d761e6b4f1" path="/var/lib/kubelet/pods/75afe340-0b9c-4b4c-89ab-20d761e6b4f1/volumes" Nov 01 06:49:37.515925 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:49:37.515910 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adbac784-4ded-4ca3-b985-3e59983fc637" path="/var/lib/kubelet/pods/adbac784-4ded-4ca3-b985-3e59983fc637/volumes" Nov 01 06:49:39.973367 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:49:39.973328 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:50:06.507666 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:06.507627 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv"] Nov 01 06:50:06.508370 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:06.507913 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" podUID="465cf21e-32b3-478b-9a76-99f363d61d2f" containerName="istio-proxy" containerID="cri-o://29841af0a325ea9554082c995de40c82b9b97b8f35adc35617f8e57518d426b1" gracePeriod=30 Nov 01 06:50:09.972421 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:50:09.972373 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:50:11.263771 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.263730 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7125-deca3b"] Nov 01 06:50:11.264226 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.264204 5040 cpu_manager.go:414] "containerMap: removing container" podUID="adbac784-4ded-4ca3-b985-3e59983fc637" containerName="main" Nov 01 06:50:11.264226 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.264227 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adbac784-4ded-4ca3-b985-3e59983fc637" containerName="main" Nov 01 06:50:11.264358 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.264245 5040 cpu_manager.go:414] "containerMap: removing container" podUID="ac0d876d-4d31-4b49-b5ed-98dce05ee7a0" containerName="revoke-keys" Nov 01 06:50:11.264358 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.264250 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac0d876d-4d31-4b49-b5ed-98dce05ee7a0" containerName="revoke-keys" Nov 01 06:50:11.264358 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.264279 5040 cpu_manager.go:414] "containerMap: removing container" podUID="57cf5e07-36de-48cf-b938-138484cea3a6" containerName="main" Nov 01 06:50:11.264358 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.264284 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57cf5e07-36de-48cf-b938-138484cea3a6" containerName="main" Nov 01 06:50:11.264358 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.264297 5040 cpu_manager.go:414] "containerMap: removing container" podUID="75afe340-0b9c-4b4c-89ab-20d761e6b4f1" containerName="main" Nov 01 06:50:11.264358 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.264301 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75afe340-0b9c-4b4c-89ab-20d761e6b4f1" containerName="main" Nov 01 06:50:11.264358 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.264308 5040 cpu_manager.go:414] "containerMap: removing container" podUID="1878b6fb-f8fe-4f4f-8f4e-056c66fc206a" containerName="main" Nov 01 06:50:11.264358 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.264313 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1878b6fb-f8fe-4f4f-8f4e-056c66fc206a" containerName="main" Nov 01 06:50:11.264652 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.264404 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="57cf5e07-36de-48cf-b938-138484cea3a6" containerName="main" Nov 01 06:50:11.264652 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.264413 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="75afe340-0b9c-4b4c-89ab-20d761e6b4f1" containerName="main" Nov 01 06:50:11.264652 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.264419 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="1878b6fb-f8fe-4f4f-8f4e-056c66fc206a" containerName="main" Nov 01 06:50:11.264652 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.264427 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="adbac784-4ded-4ca3-b985-3e59983fc637" containerName="main" Nov 01 06:50:11.264652 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.264434 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac0d876d-4d31-4b49-b5ed-98dce05ee7a0" containerName="revoke-keys" Nov 01 06:50:11.267369 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.267353 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7125-deca3b" Nov 01 06:50:11.320780 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.320748 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7125-deca3b"] Nov 01 06:50:11.367909 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.367877 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hgn5l\"" Nov 01 06:50:11.410887 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.410848 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgm7z\" (UniqueName: \"kubernetes.io/projected/047831ef-da79-4bce-b2e1-e71bdfaf3533-kube-api-access-jgm7z\") pod \"test-curl-7125-deca3b\" (UID: \"047831ef-da79-4bce-b2e1-e71bdfaf3533\") " pod="opendatahub/test-curl-7125-deca3b" Nov 01 06:50:11.511578 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.511544 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgm7z\" (UniqueName: \"kubernetes.io/projected/047831ef-da79-4bce-b2e1-e71bdfaf3533-kube-api-access-jgm7z\") pod \"test-curl-7125-deca3b\" (UID: \"047831ef-da79-4bce-b2e1-e71bdfaf3533\") " pod="opendatahub/test-curl-7125-deca3b" Nov 01 06:50:11.573134 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.573100 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgm7z\" (UniqueName: \"kubernetes.io/projected/047831ef-da79-4bce-b2e1-e71bdfaf3533-kube-api-access-jgm7z\") pod \"test-curl-7125-deca3b\" (UID: \"047831ef-da79-4bce-b2e1-e71bdfaf3533\") " pod="opendatahub/test-curl-7125-deca3b" Nov 01 06:50:11.580030 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.580001 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7125-deca3b" Nov 01 06:50:11.802940 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.802911 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" Nov 01 06:50:11.915401 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.915362 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/465cf21e-32b3-478b-9a76-99f363d61d2f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/465cf21e-32b3-478b-9a76-99f363d61d2f-credential-socket\") pod \"465cf21e-32b3-478b-9a76-99f363d61d2f\" (UID: \"465cf21e-32b3-478b-9a76-99f363d61d2f\") " Nov 01 06:50:11.915653 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.915506 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/465cf21e-32b3-478b-9a76-99f363d61d2f-kube-api-access-lhkgq\" (UniqueName: \"kubernetes.io/projected/465cf21e-32b3-478b-9a76-99f363d61d2f-kube-api-access-lhkgq\") pod \"465cf21e-32b3-478b-9a76-99f363d61d2f\" (UID: \"465cf21e-32b3-478b-9a76-99f363d61d2f\") " Nov 01 06:50:11.915653 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.915556 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/465cf21e-32b3-478b-9a76-99f363d61d2f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/465cf21e-32b3-478b-9a76-99f363d61d2f-istio-podinfo\") pod \"465cf21e-32b3-478b-9a76-99f363d61d2f\" (UID: \"465cf21e-32b3-478b-9a76-99f363d61d2f\") " Nov 01 06:50:11.915653 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.915606 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/465cf21e-32b3-478b-9a76-99f363d61d2f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/465cf21e-32b3-478b-9a76-99f363d61d2f-istio-data\") pod \"465cf21e-32b3-478b-9a76-99f363d61d2f\" (UID: \"465cf21e-32b3-478b-9a76-99f363d61d2f\") " Nov 01 06:50:11.915856 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.915670 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/465cf21e-32b3-478b-9a76-99f363d61d2f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/465cf21e-32b3-478b-9a76-99f363d61d2f-workload-socket\") pod \"465cf21e-32b3-478b-9a76-99f363d61d2f\" (UID: \"465cf21e-32b3-478b-9a76-99f363d61d2f\") " Nov 01 06:50:11.915856 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.915713 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/465cf21e-32b3-478b-9a76-99f363d61d2f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/465cf21e-32b3-478b-9a76-99f363d61d2f-workload-certs\") pod \"465cf21e-32b3-478b-9a76-99f363d61d2f\" (UID: \"465cf21e-32b3-478b-9a76-99f363d61d2f\") " Nov 01 06:50:11.915856 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.915742 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/465cf21e-32b3-478b-9a76-99f363d61d2f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/465cf21e-32b3-478b-9a76-99f363d61d2f-istiod-ca-cert\") pod \"465cf21e-32b3-478b-9a76-99f363d61d2f\" (UID: \"465cf21e-32b3-478b-9a76-99f363d61d2f\") " Nov 01 06:50:11.915856 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.915758 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465cf21e-32b3-478b-9a76-99f363d61d2f-credential-socket" pod "465cf21e-32b3-478b-9a76-99f363d61d2f" (UID: "465cf21e-32b3-478b-9a76-99f363d61d2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:50:11.915856 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.915786 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/465cf21e-32b3-478b-9a76-99f363d61d2f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/465cf21e-32b3-478b-9a76-99f363d61d2f-istio-envoy\") pod \"465cf21e-32b3-478b-9a76-99f363d61d2f\" (UID: \"465cf21e-32b3-478b-9a76-99f363d61d2f\") " Nov 01 06:50:11.915856 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.915837 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/465cf21e-32b3-478b-9a76-99f363d61d2f-istio-token\" (UniqueName: \"kubernetes.io/projected/465cf21e-32b3-478b-9a76-99f363d61d2f-istio-token\") pod \"465cf21e-32b3-478b-9a76-99f363d61d2f\" (UID: \"465cf21e-32b3-478b-9a76-99f363d61d2f\") " Nov 01 06:50:11.916587 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.916169 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465cf21e-32b3-478b-9a76-99f363d61d2f-workload-socket" pod "465cf21e-32b3-478b-9a76-99f363d61d2f" (UID: "465cf21e-32b3-478b-9a76-99f363d61d2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:50:11.916587 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.916370 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/465cf21e-32b3-478b-9a76-99f363d61d2f-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:11.916587 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.916448 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/465cf21e-32b3-478b-9a76-99f363d61d2f-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:11.916587 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.916520 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465cf21e-32b3-478b-9a76-99f363d61d2f-istio-data" pod "465cf21e-32b3-478b-9a76-99f363d61d2f" (UID: "465cf21e-32b3-478b-9a76-99f363d61d2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:50:11.916587 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.916552 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/465cf21e-32b3-478b-9a76-99f363d61d2f-istiod-ca-cert" pod "465cf21e-32b3-478b-9a76-99f363d61d2f" (UID: "465cf21e-32b3-478b-9a76-99f363d61d2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:50:11.916587 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.916587 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465cf21e-32b3-478b-9a76-99f363d61d2f-workload-certs" pod "465cf21e-32b3-478b-9a76-99f363d61d2f" (UID: "465cf21e-32b3-478b-9a76-99f363d61d2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:50:11.918460 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.918433 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/465cf21e-32b3-478b-9a76-99f363d61d2f-kube-api-access-lhkgq" pod "465cf21e-32b3-478b-9a76-99f363d61d2f" (UID: "465cf21e-32b3-478b-9a76-99f363d61d2f"). InnerVolumeSpecName "kube-api-access-lhkgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:50:11.918743 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.918719 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/465cf21e-32b3-478b-9a76-99f363d61d2f-istio-token" pod "465cf21e-32b3-478b-9a76-99f363d61d2f" (UID: "465cf21e-32b3-478b-9a76-99f363d61d2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:50:11.918826 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.918759 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/465cf21e-32b3-478b-9a76-99f363d61d2f-istio-podinfo" pod "465cf21e-32b3-478b-9a76-99f363d61d2f" (UID: "465cf21e-32b3-478b-9a76-99f363d61d2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:50:11.918876 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.918856 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465cf21e-32b3-478b-9a76-99f363d61d2f-istio-envoy" pod "465cf21e-32b3-478b-9a76-99f363d61d2f" (UID: "465cf21e-32b3-478b-9a76-99f363d61d2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:50:11.959382 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:11.959349 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7125-deca3b"] Nov 01 06:50:11.960011 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:50:11.959984 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod047831ef_da79_4bce_b2e1_e71bdfaf3533.slice/crio-1575b843d04a7b67c1395e8d668f1051226b5fd2a53cf171f5b278e99df2cb79 WatchSource:0}: Error finding container 1575b843d04a7b67c1395e8d668f1051226b5fd2a53cf171f5b278e99df2cb79: Status 404 returned error can't find the container with id 1575b843d04a7b67c1395e8d668f1051226b5fd2a53cf171f5b278e99df2cb79 Nov 01 06:50:12.017254 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:12.017218 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhkgq\" (UniqueName: \"kubernetes.io/projected/465cf21e-32b3-478b-9a76-99f363d61d2f-kube-api-access-lhkgq\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:12.017425 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:12.017260 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/465cf21e-32b3-478b-9a76-99f363d61d2f-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:12.017425 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:12.017276 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/465cf21e-32b3-478b-9a76-99f363d61d2f-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:12.017425 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:12.017292 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/465cf21e-32b3-478b-9a76-99f363d61d2f-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:12.017425 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:12.017307 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/465cf21e-32b3-478b-9a76-99f363d61d2f-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:12.017425 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:12.017322 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/465cf21e-32b3-478b-9a76-99f363d61d2f-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:12.017425 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:12.017335 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/465cf21e-32b3-478b-9a76-99f363d61d2f-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:12.549915 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:12.549881 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7125-deca3b" event={"ID":"047831ef-da79-4bce-b2e1-e71bdfaf3533","Type":"ContainerStarted","Data":"db529dba8da6a48028bc431bd63022112442000361a96fbc1b0305a67a6a9844"} Nov 01 06:50:12.550325 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:12.549928 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7125-deca3b" event={"ID":"047831ef-da79-4bce-b2e1-e71bdfaf3533","Type":"ContainerStarted","Data":"1575b843d04a7b67c1395e8d668f1051226b5fd2a53cf171f5b278e99df2cb79"} Nov 01 06:50:12.551388 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:12.551362 5040 generic.go:356] "Generic (PLEG): container finished" podID="465cf21e-32b3-478b-9a76-99f363d61d2f" containerID="29841af0a325ea9554082c995de40c82b9b97b8f35adc35617f8e57518d426b1" exitCode=0 Nov 01 06:50:12.551522 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:12.551401 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" Nov 01 06:50:12.551522 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:12.551442 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" event={"ID":"465cf21e-32b3-478b-9a76-99f363d61d2f","Type":"ContainerDied","Data":"29841af0a325ea9554082c995de40c82b9b97b8f35adc35617f8e57518d426b1"} Nov 01 06:50:12.551522 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:12.551512 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv" event={"ID":"465cf21e-32b3-478b-9a76-99f363d61d2f","Type":"ContainerDied","Data":"c235546850d77e67dfb0bc6ec39c31631b164edbcd0d74cf88115f7c00157b6c"} Nov 01 06:50:12.551631 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:12.551536 5040 scope.go:122] "RemoveContainer" containerID="29841af0a325ea9554082c995de40c82b9b97b8f35adc35617f8e57518d426b1" Nov 01 06:50:12.562460 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:12.562440 5040 scope.go:122] "RemoveContainer" containerID="29841af0a325ea9554082c995de40c82b9b97b8f35adc35617f8e57518d426b1" Nov 01 06:50:12.562770 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:50:12.562752 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29841af0a325ea9554082c995de40c82b9b97b8f35adc35617f8e57518d426b1\": container with ID starting with 29841af0a325ea9554082c995de40c82b9b97b8f35adc35617f8e57518d426b1 not found: ID does not exist" containerID="29841af0a325ea9554082c995de40c82b9b97b8f35adc35617f8e57518d426b1" Nov 01 06:50:12.562822 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:12.562779 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29841af0a325ea9554082c995de40c82b9b97b8f35adc35617f8e57518d426b1"} err="failed to get container status \"29841af0a325ea9554082c995de40c82b9b97b8f35adc35617f8e57518d426b1\": rpc error: code = NotFound desc = could not find container \"29841af0a325ea9554082c995de40c82b9b97b8f35adc35617f8e57518d426b1\": container with ID starting with 29841af0a325ea9554082c995de40c82b9b97b8f35adc35617f8e57518d426b1 not found: ID does not exist" Nov 01 06:50:12.666251 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:12.666193 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7125-deca3b" podStartSLOduration=1.666177626 podStartE2EDuration="1.666177626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:50:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:50:12.663868032 +0000 UTC m=+3891.318985200" watchObservedRunningTime="2026-11-01 06:50:12.666177626 +0000 UTC m=+3891.321294792" Nov 01 06:50:12.778146 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:12.778108 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv"] Nov 01 06:50:12.787488 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:12.787438 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4c9sv"] Nov 01 06:50:13.514379 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:13.514347 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="465cf21e-32b3-478b-9a76-99f363d61d2f" path="/var/lib/kubelet/pods/465cf21e-32b3-478b-9a76-99f363d61d2f/volumes" Nov 01 06:50:14.148855 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:14.148815 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7125-deca3b"] Nov 01 06:50:14.155260 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:14.155233 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7125-deca3b"] Nov 01 06:50:14.303921 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:14.303888 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7125-e7d4b6"] Nov 01 06:50:14.304395 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:14.304379 5040 cpu_manager.go:414] "containerMap: removing container" podUID="047831ef-da79-4bce-b2e1-e71bdfaf3533" containerName="test-curl-7125-deca3b" Nov 01 06:50:14.304457 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:14.304395 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="047831ef-da79-4bce-b2e1-e71bdfaf3533" containerName="test-curl-7125-deca3b" Nov 01 06:50:14.304457 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:14.304409 5040 cpu_manager.go:414] "containerMap: removing container" podUID="465cf21e-32b3-478b-9a76-99f363d61d2f" containerName="istio-proxy" Nov 01 06:50:14.304457 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:14.304414 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="465cf21e-32b3-478b-9a76-99f363d61d2f" containerName="istio-proxy" Nov 01 06:50:14.304590 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:14.304514 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="047831ef-da79-4bce-b2e1-e71bdfaf3533" containerName="test-curl-7125-deca3b" Nov 01 06:50:14.304590 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:14.304524 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="465cf21e-32b3-478b-9a76-99f363d61d2f" containerName="istio-proxy" Nov 01 06:50:14.309125 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:14.309105 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7125-e7d4b6" Nov 01 06:50:14.337646 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:14.337616 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czsj9\" (UniqueName: \"kubernetes.io/projected/76b2259e-9e3e-4636-8dd8-883388a13501-kube-api-access-czsj9\") pod \"test-curl-7125-e7d4b6\" (UID: \"76b2259e-9e3e-4636-8dd8-883388a13501\") " pod="opendatahub/test-curl-7125-e7d4b6" Nov 01 06:50:14.363570 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:14.363541 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7125-e7d4b6"] Nov 01 06:50:14.438146 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:14.438055 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czsj9\" (UniqueName: \"kubernetes.io/projected/76b2259e-9e3e-4636-8dd8-883388a13501-kube-api-access-czsj9\") pod \"test-curl-7125-e7d4b6\" (UID: \"76b2259e-9e3e-4636-8dd8-883388a13501\") " pod="opendatahub/test-curl-7125-e7d4b6" Nov 01 06:50:14.499850 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:14.499813 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czsj9\" (UniqueName: \"kubernetes.io/projected/76b2259e-9e3e-4636-8dd8-883388a13501-kube-api-access-czsj9\") pod \"test-curl-7125-e7d4b6\" (UID: \"76b2259e-9e3e-4636-8dd8-883388a13501\") " pod="opendatahub/test-curl-7125-e7d4b6" Nov 01 06:50:14.567567 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:14.567528 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7125-deca3b" podUID="047831ef-da79-4bce-b2e1-e71bdfaf3533" containerName="test-curl-7125-deca3b" containerID="cri-o://db529dba8da6a48028bc431bd63022112442000361a96fbc1b0305a67a6a9844" gracePeriod=2 Nov 01 06:50:14.619666 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:50:14.619621 5040 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7125-deca3b\" is forbidden: User \"system:node:ip-10-0-143-195.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-195.us-west-1.compute.internal' and this object" podUID="047831ef-da79-4bce-b2e1-e71bdfaf3533" pod="opendatahub/test-curl-7125-deca3b" Nov 01 06:50:14.620713 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:14.620693 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7125-e7d4b6" Nov 01 06:50:14.994878 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:14.994847 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7125-e7d4b6"] Nov 01 06:50:14.995374 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:50:14.995336 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76b2259e_9e3e_4636_8dd8_883388a13501.slice/crio-a0761a6f184988ef2c4d708f1756e5fa4c7a3fb8eb240cb9a6c044d891a85a55 WatchSource:0}: Error finding container a0761a6f184988ef2c4d708f1756e5fa4c7a3fb8eb240cb9a6c044d891a85a55: Status 404 returned error can't find the container with id a0761a6f184988ef2c4d708f1756e5fa4c7a3fb8eb240cb9a6c044d891a85a55 Nov 01 06:50:15.575569 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:15.575531 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7125-e7d4b6" event={"ID":"76b2259e-9e3e-4636-8dd8-883388a13501","Type":"ContainerStarted","Data":"5018f3056a935cdc7c594bbb5c4f92028f3728124218dacd2881ee3d4ef09a6d"} Nov 01 06:50:15.575569 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:15.575575 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7125-e7d4b6" event={"ID":"76b2259e-9e3e-4636-8dd8-883388a13501","Type":"ContainerStarted","Data":"a0761a6f184988ef2c4d708f1756e5fa4c7a3fb8eb240cb9a6c044d891a85a55"} Nov 01 06:50:15.690404 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:15.690346 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7125-e7d4b6" podStartSLOduration=1.6903311090000002 podStartE2EDuration="1.690331109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:50:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:50:15.689408232 +0000 UTC m=+3894.344525401" watchObservedRunningTime="2026-11-01 06:50:15.690331109 +0000 UTC m=+3894.345448275" Nov 01 06:50:16.803959 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:16.803927 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7125-deca3b" Nov 01 06:50:16.858759 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:16.858663 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/047831ef-da79-4bce-b2e1-e71bdfaf3533-kube-api-access-jgm7z\" (UniqueName: \"kubernetes.io/projected/047831ef-da79-4bce-b2e1-e71bdfaf3533-kube-api-access-jgm7z\") pod \"047831ef-da79-4bce-b2e1-e71bdfaf3533\" (UID: \"047831ef-da79-4bce-b2e1-e71bdfaf3533\") " Nov 01 06:50:16.860801 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:16.860772 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/047831ef-da79-4bce-b2e1-e71bdfaf3533-kube-api-access-jgm7z" pod "047831ef-da79-4bce-b2e1-e71bdfaf3533" (UID: "047831ef-da79-4bce-b2e1-e71bdfaf3533"). InnerVolumeSpecName "kube-api-access-jgm7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:50:16.959946 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:16.959909 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgm7z\" (UniqueName: \"kubernetes.io/projected/047831ef-da79-4bce-b2e1-e71bdfaf3533-kube-api-access-jgm7z\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:17.101692 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:17.101642 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7125-e7d4b6"] Nov 01 06:50:17.105689 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:17.105657 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7125-e7d4b6"] Nov 01 06:50:17.514239 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:17.514205 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="047831ef-da79-4bce-b2e1-e71bdfaf3533" path="/var/lib/kubelet/pods/047831ef-da79-4bce-b2e1-e71bdfaf3533/volumes" Nov 01 06:50:17.590946 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:17.590908 5040 generic.go:356] "Generic (PLEG): container finished" podID="047831ef-da79-4bce-b2e1-e71bdfaf3533" containerID="db529dba8da6a48028bc431bd63022112442000361a96fbc1b0305a67a6a9844" exitCode=137 Nov 01 06:50:17.591155 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:17.591017 5040 scope.go:122] "RemoveContainer" containerID="db529dba8da6a48028bc431bd63022112442000361a96fbc1b0305a67a6a9844" Nov 01 06:50:17.591155 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:17.591035 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7125-deca3b" Nov 01 06:50:17.591155 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:17.591073 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7125-e7d4b6" podUID="76b2259e-9e3e-4636-8dd8-883388a13501" containerName="test-curl-7125-e7d4b6" containerID="cri-o://5018f3056a935cdc7c594bbb5c4f92028f3728124218dacd2881ee3d4ef09a6d" gracePeriod=2 Nov 01 06:50:17.602931 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:17.602903 5040 scope.go:122] "RemoveContainer" containerID="db529dba8da6a48028bc431bd63022112442000361a96fbc1b0305a67a6a9844" Nov 01 06:50:17.603305 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:50:17.603254 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db529dba8da6a48028bc431bd63022112442000361a96fbc1b0305a67a6a9844\": container with ID starting with db529dba8da6a48028bc431bd63022112442000361a96fbc1b0305a67a6a9844 not found: ID does not exist" containerID="db529dba8da6a48028bc431bd63022112442000361a96fbc1b0305a67a6a9844" Nov 01 06:50:17.603390 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:17.603295 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db529dba8da6a48028bc431bd63022112442000361a96fbc1b0305a67a6a9844"} err="failed to get container status \"db529dba8da6a48028bc431bd63022112442000361a96fbc1b0305a67a6a9844\": rpc error: code = NotFound desc = could not find container \"db529dba8da6a48028bc431bd63022112442000361a96fbc1b0305a67a6a9844\": container with ID starting with db529dba8da6a48028bc431bd63022112442000361a96fbc1b0305a67a6a9844 not found: ID does not exist" Nov 01 06:50:18.157080 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:18.157033 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e43d1ae3-nglck"] Nov 01 06:50:18.157738 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:18.157714 5040 cpu_manager.go:414] "containerMap: removing container" podUID="76b2259e-9e3e-4636-8dd8-883388a13501" containerName="test-curl-7125-e7d4b6" Nov 01 06:50:18.157851 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:18.157739 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76b2259e-9e3e-4636-8dd8-883388a13501" containerName="test-curl-7125-e7d4b6" Nov 01 06:50:18.157919 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:18.157909 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="76b2259e-9e3e-4636-8dd8-883388a13501" containerName="test-curl-7125-e7d4b6" Nov 01 06:50:18.162390 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:18.162367 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43d1ae3-nglck" Nov 01 06:50:18.172472 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:18.172442 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c3da87f-7ffc-4057-a412-80a547f8099b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e43d1ae3-nglck\" (UID: \"8c3da87f-7ffc-4057-a412-80a547f8099b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43d1ae3-nglck" Nov 01 06:50:18.172472 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:18.172492 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r577\" (UniqueName: \"kubernetes.io/projected/8c3da87f-7ffc-4057-a412-80a547f8099b-kube-api-access-7r577\") pod \"maas-api-revoke-sub-e43d1ae3-nglck\" (UID: \"8c3da87f-7ffc-4057-a412-80a547f8099b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43d1ae3-nglck" Nov 01 06:50:18.172668 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:18.172638 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c3da87f-7ffc-4057-a412-80a547f8099b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e43d1ae3-nglck\" (UID: \"8c3da87f-7ffc-4057-a412-80a547f8099b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43d1ae3-nglck" Nov 01 06:50:18.218101 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:18.218064 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e43d1ae3-nglck"] Nov 01 06:50:18.264981 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:18.264937 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:50:18.265190 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:18.264994 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:50:18.265190 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:18.265054 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:50:18.265854 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:18.265835 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:50:18.273417 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:18.273389 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c3da87f-7ffc-4057-a412-80a547f8099b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e43d1ae3-nglck\" (UID: \"8c3da87f-7ffc-4057-a412-80a547f8099b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43d1ae3-nglck" Nov 01 06:50:18.273560 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:18.273541 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c3da87f-7ffc-4057-a412-80a547f8099b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e43d1ae3-nglck\" (UID: \"8c3da87f-7ffc-4057-a412-80a547f8099b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43d1ae3-nglck" Nov 01 06:50:18.273654 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:18.273582 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r577\" (UniqueName: \"kubernetes.io/projected/8c3da87f-7ffc-4057-a412-80a547f8099b-kube-api-access-7r577\") pod \"maas-api-revoke-sub-e43d1ae3-nglck\" (UID: \"8c3da87f-7ffc-4057-a412-80a547f8099b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43d1ae3-nglck" Nov 01 06:50:18.274248 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:18.274226 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c3da87f-7ffc-4057-a412-80a547f8099b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e43d1ae3-nglck\" (UID: \"8c3da87f-7ffc-4057-a412-80a547f8099b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43d1ae3-nglck" Nov 01 06:50:18.275994 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:18.275974 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c3da87f-7ffc-4057-a412-80a547f8099b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e43d1ae3-nglck\" (UID: \"8c3da87f-7ffc-4057-a412-80a547f8099b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43d1ae3-nglck" Nov 01 06:50:18.335223 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:18.335193 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r577\" (UniqueName: \"kubernetes.io/projected/8c3da87f-7ffc-4057-a412-80a547f8099b-kube-api-access-7r577\") pod \"maas-api-revoke-sub-e43d1ae3-nglck\" (UID: \"8c3da87f-7ffc-4057-a412-80a547f8099b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43d1ae3-nglck" Nov 01 06:50:18.475699 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:18.475604 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43d1ae3-nglck" Nov 01 06:50:18.861389 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:18.861356 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e43d1ae3-nglck"] Nov 01 06:50:18.861724 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:50:18.861695 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c3da87f_7ffc_4057_a412_80a547f8099b.slice/crio-c3601d72b7a2e593c8dc1afaf3294e319355e99900bb054c37bc244e7e5c8765 WatchSource:0}: Error finding container c3601d72b7a2e593c8dc1afaf3294e319355e99900bb054c37bc244e7e5c8765: Status 404 returned error can't find the container with id c3601d72b7a2e593c8dc1afaf3294e319355e99900bb054c37bc244e7e5c8765 Nov 01 06:50:19.610152 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:19.610118 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43d1ae3-nglck" event={"ID":"8c3da87f-7ffc-4057-a412-80a547f8099b","Type":"ContainerStarted","Data":"9f84efb1a2bf597369e371f5f8217ee8dcea1d8ce387bba4fe1bf66772e2d449"} Nov 01 06:50:19.610460 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:19.610172 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43d1ae3-nglck" event={"ID":"8c3da87f-7ffc-4057-a412-80a547f8099b","Type":"ContainerStarted","Data":"c3601d72b7a2e593c8dc1afaf3294e319355e99900bb054c37bc244e7e5c8765"} Nov 01 06:50:19.727782 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:19.727724 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43d1ae3-nglck" podStartSLOduration=1.727710732 podStartE2EDuration="1.727710732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:50:19.726792151 +0000 UTC m=+3898.381909319" watchObservedRunningTime="2026-11-01 06:50:19.727710732 +0000 UTC m=+3898.382827962" Nov 01 06:50:19.859034 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:19.859007 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7125-e7d4b6" Nov 01 06:50:19.891598 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:19.891552 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76b2259e-9e3e-4636-8dd8-883388a13501-kube-api-access-czsj9\" (UniqueName: \"kubernetes.io/projected/76b2259e-9e3e-4636-8dd8-883388a13501-kube-api-access-czsj9\") pod \"76b2259e-9e3e-4636-8dd8-883388a13501\" (UID: \"76b2259e-9e3e-4636-8dd8-883388a13501\") " Nov 01 06:50:19.893636 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:19.893606 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76b2259e-9e3e-4636-8dd8-883388a13501-kube-api-access-czsj9" pod "76b2259e-9e3e-4636-8dd8-883388a13501" (UID: "76b2259e-9e3e-4636-8dd8-883388a13501"). InnerVolumeSpecName "kube-api-access-czsj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:50:19.992537 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:19.992464 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czsj9\" (UniqueName: \"kubernetes.io/projected/76b2259e-9e3e-4636-8dd8-883388a13501-kube-api-access-czsj9\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:20.618652 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:20.618611 5040 generic.go:356] "Generic (PLEG): container finished" podID="8c3da87f-7ffc-4057-a412-80a547f8099b" containerID="9f84efb1a2bf597369e371f5f8217ee8dcea1d8ce387bba4fe1bf66772e2d449" exitCode=0 Nov 01 06:50:20.619123 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:20.618687 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43d1ae3-nglck" event={"ID":"8c3da87f-7ffc-4057-a412-80a547f8099b","Type":"ContainerDied","Data":"9f84efb1a2bf597369e371f5f8217ee8dcea1d8ce387bba4fe1bf66772e2d449"} Nov 01 06:50:20.620176 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:20.620150 5040 generic.go:356] "Generic (PLEG): container finished" podID="76b2259e-9e3e-4636-8dd8-883388a13501" containerID="5018f3056a935cdc7c594bbb5c4f92028f3728124218dacd2881ee3d4ef09a6d" exitCode=137 Nov 01 06:50:20.620284 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:20.620186 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7125-e7d4b6" Nov 01 06:50:20.620284 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:20.620216 5040 scope.go:122] "RemoveContainer" containerID="5018f3056a935cdc7c594bbb5c4f92028f3728124218dacd2881ee3d4ef09a6d" Nov 01 06:50:20.633210 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:20.633182 5040 scope.go:122] "RemoveContainer" containerID="5018f3056a935cdc7c594bbb5c4f92028f3728124218dacd2881ee3d4ef09a6d" Nov 01 06:50:20.633591 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:50:20.633570 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5018f3056a935cdc7c594bbb5c4f92028f3728124218dacd2881ee3d4ef09a6d\": container with ID starting with 5018f3056a935cdc7c594bbb5c4f92028f3728124218dacd2881ee3d4ef09a6d not found: ID does not exist" containerID="5018f3056a935cdc7c594bbb5c4f92028f3728124218dacd2881ee3d4ef09a6d" Nov 01 06:50:20.633659 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:20.633602 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5018f3056a935cdc7c594bbb5c4f92028f3728124218dacd2881ee3d4ef09a6d"} err="failed to get container status \"5018f3056a935cdc7c594bbb5c4f92028f3728124218dacd2881ee3d4ef09a6d\": rpc error: code = NotFound desc = could not find container \"5018f3056a935cdc7c594bbb5c4f92028f3728124218dacd2881ee3d4ef09a6d\": container with ID starting with 5018f3056a935cdc7c594bbb5c4f92028f3728124218dacd2881ee3d4ef09a6d not found: ID does not exist" Nov 01 06:50:20.789101 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:50:20.789055 5040 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7125-e7d4b6\" is forbidden: User \"system:node:ip-10-0-143-195.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-195.us-west-1.compute.internal' and this object" podUID="76b2259e-9e3e-4636-8dd8-883388a13501" pod="opendatahub/test-curl-7125-e7d4b6" Nov 01 06:50:21.515902 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:21.515867 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76b2259e-9e3e-4636-8dd8-883388a13501" path="/var/lib/kubelet/pods/76b2259e-9e3e-4636-8dd8-883388a13501/volumes" Nov 01 06:50:21.562654 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:50:21.562617 5040 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7125-e7d4b6\" is forbidden: User \"system:node:ip-10-0-143-195.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-195.us-west-1.compute.internal' and this object" podUID="76b2259e-9e3e-4636-8dd8-883388a13501" pod="opendatahub/test-curl-7125-e7d4b6" Nov 01 06:50:21.769948 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:21.769884 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43d1ae3-nglck" Nov 01 06:50:21.808627 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:21.808597 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c3da87f-7ffc-4057-a412-80a547f8099b-kube-api-access-7r577\" (UniqueName: \"kubernetes.io/projected/8c3da87f-7ffc-4057-a412-80a547f8099b-kube-api-access-7r577\") pod \"8c3da87f-7ffc-4057-a412-80a547f8099b\" (UID: \"8c3da87f-7ffc-4057-a412-80a547f8099b\") " Nov 01 06:50:21.808812 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:21.808655 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c3da87f-7ffc-4057-a412-80a547f8099b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c3da87f-7ffc-4057-a412-80a547f8099b-maas-api-openshift-ca\") pod \"8c3da87f-7ffc-4057-a412-80a547f8099b\" (UID: \"8c3da87f-7ffc-4057-a412-80a547f8099b\") " Nov 01 06:50:21.808812 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:21.808766 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c3da87f-7ffc-4057-a412-80a547f8099b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c3da87f-7ffc-4057-a412-80a547f8099b-maas-api-xks-ca\") pod \"8c3da87f-7ffc-4057-a412-80a547f8099b\" (UID: \"8c3da87f-7ffc-4057-a412-80a547f8099b\") " Nov 01 06:50:21.809022 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:21.808996 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c3da87f-7ffc-4057-a412-80a547f8099b-maas-api-openshift-ca" pod "8c3da87f-7ffc-4057-a412-80a547f8099b" (UID: "8c3da87f-7ffc-4057-a412-80a547f8099b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:50:21.810791 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:21.810767 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c3da87f-7ffc-4057-a412-80a547f8099b-kube-api-access-7r577" pod "8c3da87f-7ffc-4057-a412-80a547f8099b" (UID: "8c3da87f-7ffc-4057-a412-80a547f8099b"). InnerVolumeSpecName "kube-api-access-7r577". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:50:21.810932 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:21.810909 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c3da87f-7ffc-4057-a412-80a547f8099b-maas-api-xks-ca" pod "8c3da87f-7ffc-4057-a412-80a547f8099b" (UID: "8c3da87f-7ffc-4057-a412-80a547f8099b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:50:21.909903 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:21.909856 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r577\" (UniqueName: \"kubernetes.io/projected/8c3da87f-7ffc-4057-a412-80a547f8099b-kube-api-access-7r577\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:21.909903 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:21.909892 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c3da87f-7ffc-4057-a412-80a547f8099b-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:21.909903 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:21.909902 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c3da87f-7ffc-4057-a412-80a547f8099b-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:22.248522 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:22.248463 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 06:50:22.254248 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:22.254221 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 06:50:22.634947 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:22.634850 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43d1ae3-nglck" Nov 01 06:50:22.634947 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:22.634871 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43d1ae3-nglck" event={"ID":"8c3da87f-7ffc-4057-a412-80a547f8099b","Type":"ContainerDied","Data":"c3601d72b7a2e593c8dc1afaf3294e319355e99900bb054c37bc244e7e5c8765"} Nov 01 06:50:22.634947 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:22.634917 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3601d72b7a2e593c8dc1afaf3294e319355e99900bb054c37bc244e7e5c8765" Nov 01 06:50:23.951666 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:23.951632 5040 scope.go:122] "RemoveContainer" containerID="c946728d0cee3e23c124f2a70ca2d8d302c1b0d15aca1ac76fc1ac40c2da9d82" Nov 01 06:50:23.961124 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:23.961101 5040 scope.go:122] "RemoveContainer" containerID="949a6d1f21f5079d7114a9e2768b0c51c0a685052b5e7668119242c365915d8b" Nov 01 06:50:23.969987 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:23.969962 5040 scope.go:122] "RemoveContainer" containerID="34474f24678aa795d9ca8b0e988943df18e5928f67e2340257a3760266ae1a24" Nov 01 06:50:23.978332 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:23.978314 5040 scope.go:122] "RemoveContainer" containerID="dd00248788b40a58b0244afaa32709485526feaf81eb0fc7f6090c0bcaed951f" Nov 01 06:50:28.944796 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:28.944750 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2dd2f7d-96s4m"] Nov 01 06:50:28.945418 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:28.945395 5040 cpu_manager.go:414] "containerMap: removing container" podUID="8c3da87f-7ffc-4057-a412-80a547f8099b" containerName="revoke-keys" Nov 01 06:50:28.945529 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:28.945421 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c3da87f-7ffc-4057-a412-80a547f8099b" containerName="revoke-keys" Nov 01 06:50:28.945609 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:28.945593 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c3da87f-7ffc-4057-a412-80a547f8099b" containerName="revoke-keys" Nov 01 06:50:28.948643 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:28.948620 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2dd2f7d-96s4m" Nov 01 06:50:28.981944 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:28.981901 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztphh\" (UniqueName: \"kubernetes.io/projected/3460d306-df7a-4fcd-93ba-0bb4977553e5-kube-api-access-ztphh\") pod \"maas-api-revoke-sub-b2dd2f7d-96s4m\" (UID: \"3460d306-df7a-4fcd-93ba-0bb4977553e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2dd2f7d-96s4m" Nov 01 06:50:28.982123 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:28.982017 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3460d306-df7a-4fcd-93ba-0bb4977553e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2dd2f7d-96s4m\" (UID: \"3460d306-df7a-4fcd-93ba-0bb4977553e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2dd2f7d-96s4m" Nov 01 06:50:28.982123 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:28.982075 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3460d306-df7a-4fcd-93ba-0bb4977553e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2dd2f7d-96s4m\" (UID: \"3460d306-df7a-4fcd-93ba-0bb4977553e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2dd2f7d-96s4m" Nov 01 06:50:29.004572 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:29.004540 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:50:29.007274 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:29.007241 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2dd2f7d-96s4m"] Nov 01 06:50:29.053601 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:29.053563 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:50:29.054846 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:29.054822 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:50:29.054917 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:29.054889 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:50:29.083547 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:29.083510 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3460d306-df7a-4fcd-93ba-0bb4977553e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2dd2f7d-96s4m\" (UID: \"3460d306-df7a-4fcd-93ba-0bb4977553e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2dd2f7d-96s4m" Nov 01 06:50:29.083758 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:29.083592 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztphh\" (UniqueName: \"kubernetes.io/projected/3460d306-df7a-4fcd-93ba-0bb4977553e5-kube-api-access-ztphh\") pod \"maas-api-revoke-sub-b2dd2f7d-96s4m\" (UID: \"3460d306-df7a-4fcd-93ba-0bb4977553e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2dd2f7d-96s4m" Nov 01 06:50:29.083758 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:29.083645 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3460d306-df7a-4fcd-93ba-0bb4977553e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2dd2f7d-96s4m\" (UID: \"3460d306-df7a-4fcd-93ba-0bb4977553e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2dd2f7d-96s4m" Nov 01 06:50:29.084109 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:29.084086 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3460d306-df7a-4fcd-93ba-0bb4977553e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2dd2f7d-96s4m\" (UID: \"3460d306-df7a-4fcd-93ba-0bb4977553e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2dd2f7d-96s4m" Nov 01 06:50:29.086038 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:29.086019 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3460d306-df7a-4fcd-93ba-0bb4977553e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2dd2f7d-96s4m\" (UID: \"3460d306-df7a-4fcd-93ba-0bb4977553e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2dd2f7d-96s4m" Nov 01 06:50:29.149837 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:29.149804 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztphh\" (UniqueName: \"kubernetes.io/projected/3460d306-df7a-4fcd-93ba-0bb4977553e5-kube-api-access-ztphh\") pod \"maas-api-revoke-sub-b2dd2f7d-96s4m\" (UID: \"3460d306-df7a-4fcd-93ba-0bb4977553e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2dd2f7d-96s4m" Nov 01 06:50:29.262136 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:29.262097 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2dd2f7d-96s4m" Nov 01 06:50:29.346232 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:29.346198 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9856327a-ll857"] Nov 01 06:50:29.351199 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:29.351176 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9856327a-ll857" Nov 01 06:50:29.386548 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:29.386506 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndld9\" (UniqueName: \"kubernetes.io/projected/f926ff2a-42d9-4af9-a597-c8734d9e1b06-kube-api-access-ndld9\") pod \"maas-api-revoke-sub-9856327a-ll857\" (UID: \"f926ff2a-42d9-4af9-a597-c8734d9e1b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9856327a-ll857" Nov 01 06:50:29.386767 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:29.386562 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f926ff2a-42d9-4af9-a597-c8734d9e1b06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9856327a-ll857\" (UID: \"f926ff2a-42d9-4af9-a597-c8734d9e1b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9856327a-ll857" Nov 01 06:50:29.386767 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:29.386674 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f926ff2a-42d9-4af9-a597-c8734d9e1b06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9856327a-ll857\" (UID: \"f926ff2a-42d9-4af9-a597-c8734d9e1b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9856327a-ll857" Nov 01 06:50:29.410111 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:29.410079 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9856327a-ll857"] Nov 01 06:50:29.488001 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:29.487965 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndld9\" (UniqueName: \"kubernetes.io/projected/f926ff2a-42d9-4af9-a597-c8734d9e1b06-kube-api-access-ndld9\") pod \"maas-api-revoke-sub-9856327a-ll857\" (UID: \"f926ff2a-42d9-4af9-a597-c8734d9e1b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9856327a-ll857" Nov 01 06:50:29.488199 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:29.488031 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f926ff2a-42d9-4af9-a597-c8734d9e1b06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9856327a-ll857\" (UID: \"f926ff2a-42d9-4af9-a597-c8734d9e1b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9856327a-ll857" Nov 01 06:50:29.488199 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:29.488070 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f926ff2a-42d9-4af9-a597-c8734d9e1b06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9856327a-ll857\" (UID: \"f926ff2a-42d9-4af9-a597-c8734d9e1b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9856327a-ll857" Nov 01 06:50:29.488828 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:29.488805 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f926ff2a-42d9-4af9-a597-c8734d9e1b06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9856327a-ll857\" (UID: \"f926ff2a-42d9-4af9-a597-c8734d9e1b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9856327a-ll857" Nov 01 06:50:29.490548 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:29.490525 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f926ff2a-42d9-4af9-a597-c8734d9e1b06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9856327a-ll857\" (UID: \"f926ff2a-42d9-4af9-a597-c8734d9e1b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9856327a-ll857" Nov 01 06:50:29.550616 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:29.550539 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndld9\" (UniqueName: \"kubernetes.io/projected/f926ff2a-42d9-4af9-a597-c8734d9e1b06-kube-api-access-ndld9\") pod \"maas-api-revoke-sub-9856327a-ll857\" (UID: \"f926ff2a-42d9-4af9-a597-c8734d9e1b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9856327a-ll857" Nov 01 06:50:29.640858 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:29.640829 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2dd2f7d-96s4m"] Nov 01 06:50:29.641662 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:50:29.641630 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3460d306_df7a_4fcd_93ba_0bb4977553e5.slice/crio-2950ea83f60806b49b37bcc6acfd2a2ca16881ae33a99be5d5c9d8e7e62f6555 WatchSource:0}: Error finding container 2950ea83f60806b49b37bcc6acfd2a2ca16881ae33a99be5d5c9d8e7e62f6555: Status 404 returned error can't find the container with id 2950ea83f60806b49b37bcc6acfd2a2ca16881ae33a99be5d5c9d8e7e62f6555 Nov 01 06:50:29.667827 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:29.667798 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9856327a-ll857" Nov 01 06:50:29.681911 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:29.681881 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2dd2f7d-96s4m" event={"ID":"3460d306-df7a-4fcd-93ba-0bb4977553e5","Type":"ContainerStarted","Data":"2950ea83f60806b49b37bcc6acfd2a2ca16881ae33a99be5d5c9d8e7e62f6555"} Nov 01 06:50:30.045224 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:30.045196 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9856327a-ll857"] Nov 01 06:50:30.045912 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:50:30.045887 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf926ff2a_42d9_4af9_a597_c8734d9e1b06.slice/crio-dde61cb41215f249b21df038ad0b68a02ebd3179104fc298e6e8f33f1d309d93 WatchSource:0}: Error finding container dde61cb41215f249b21df038ad0b68a02ebd3179104fc298e6e8f33f1d309d93: Status 404 returned error can't find the container with id dde61cb41215f249b21df038ad0b68a02ebd3179104fc298e6e8f33f1d309d93 Nov 01 06:50:30.690617 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:30.690540 5040 generic.go:356] "Generic (PLEG): container finished" podID="3460d306-df7a-4fcd-93ba-0bb4977553e5" containerID="4dca403bcbed3a8a774dcfa3a27df021985b95355fb3480b60e4fee2c52efdaa" exitCode=0 Nov 01 06:50:30.690798 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:30.690627 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2dd2f7d-96s4m" event={"ID":"3460d306-df7a-4fcd-93ba-0bb4977553e5","Type":"ContainerDied","Data":"4dca403bcbed3a8a774dcfa3a27df021985b95355fb3480b60e4fee2c52efdaa"} Nov 01 06:50:30.692358 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:30.692323 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9856327a-ll857" event={"ID":"f926ff2a-42d9-4af9-a597-c8734d9e1b06","Type":"ContainerStarted","Data":"ec00f70688314b481a4fdd17328bf4357ee0a8d615a62b9737ff426c372b90aa"} Nov 01 06:50:30.692786 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:30.692381 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9856327a-ll857" event={"ID":"f926ff2a-42d9-4af9-a597-c8734d9e1b06","Type":"ContainerStarted","Data":"dde61cb41215f249b21df038ad0b68a02ebd3179104fc298e6e8f33f1d309d93"} Nov 01 06:50:30.926170 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:30.926109 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9856327a-ll857" podStartSLOduration=1.9260947069999999 podStartE2EDuration="1.926094707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:50:30.92421585 +0000 UTC m=+3909.579333016" watchObservedRunningTime="2026-11-01 06:50:30.926094707 +0000 UTC m=+3909.581211873" Nov 01 06:50:31.699824 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:31.699791 5040 generic.go:356] "Generic (PLEG): container finished" podID="f926ff2a-42d9-4af9-a597-c8734d9e1b06" containerID="ec00f70688314b481a4fdd17328bf4357ee0a8d615a62b9737ff426c372b90aa" exitCode=0 Nov 01 06:50:31.700245 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:31.699880 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9856327a-ll857" event={"ID":"f926ff2a-42d9-4af9-a597-c8734d9e1b06","Type":"ContainerDied","Data":"ec00f70688314b481a4fdd17328bf4357ee0a8d615a62b9737ff426c372b90aa"} Nov 01 06:50:31.847766 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:31.847737 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2dd2f7d-96s4m" Nov 01 06:50:31.911297 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:31.911257 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3460d306-df7a-4fcd-93ba-0bb4977553e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3460d306-df7a-4fcd-93ba-0bb4977553e5-maas-api-xks-ca\") pod \"3460d306-df7a-4fcd-93ba-0bb4977553e5\" (UID: \"3460d306-df7a-4fcd-93ba-0bb4977553e5\") " Nov 01 06:50:31.911527 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:31.911370 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3460d306-df7a-4fcd-93ba-0bb4977553e5-kube-api-access-ztphh\" (UniqueName: \"kubernetes.io/projected/3460d306-df7a-4fcd-93ba-0bb4977553e5-kube-api-access-ztphh\") pod \"3460d306-df7a-4fcd-93ba-0bb4977553e5\" (UID: \"3460d306-df7a-4fcd-93ba-0bb4977553e5\") " Nov 01 06:50:31.911527 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:31.911410 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3460d306-df7a-4fcd-93ba-0bb4977553e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3460d306-df7a-4fcd-93ba-0bb4977553e5-maas-api-openshift-ca\") pod \"3460d306-df7a-4fcd-93ba-0bb4977553e5\" (UID: \"3460d306-df7a-4fcd-93ba-0bb4977553e5\") " Nov 01 06:50:31.911833 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:31.911804 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3460d306-df7a-4fcd-93ba-0bb4977553e5-maas-api-openshift-ca" pod "3460d306-df7a-4fcd-93ba-0bb4977553e5" (UID: "3460d306-df7a-4fcd-93ba-0bb4977553e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:50:31.913650 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:31.913627 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3460d306-df7a-4fcd-93ba-0bb4977553e5-maas-api-xks-ca" pod "3460d306-df7a-4fcd-93ba-0bb4977553e5" (UID: "3460d306-df7a-4fcd-93ba-0bb4977553e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:50:31.913761 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:31.913644 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3460d306-df7a-4fcd-93ba-0bb4977553e5-kube-api-access-ztphh" pod "3460d306-df7a-4fcd-93ba-0bb4977553e5" (UID: "3460d306-df7a-4fcd-93ba-0bb4977553e5"). InnerVolumeSpecName "kube-api-access-ztphh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:50:32.013042 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:32.012998 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztphh\" (UniqueName: \"kubernetes.io/projected/3460d306-df7a-4fcd-93ba-0bb4977553e5-kube-api-access-ztphh\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:32.013042 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:32.013038 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3460d306-df7a-4fcd-93ba-0bb4977553e5-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:32.013397 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:32.013052 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3460d306-df7a-4fcd-93ba-0bb4977553e5-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:32.707067 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:32.707038 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2dd2f7d-96s4m" Nov 01 06:50:32.707457 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:32.707080 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2dd2f7d-96s4m" event={"ID":"3460d306-df7a-4fcd-93ba-0bb4977553e5","Type":"ContainerDied","Data":"2950ea83f60806b49b37bcc6acfd2a2ca16881ae33a99be5d5c9d8e7e62f6555"} Nov 01 06:50:32.707457 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:32.707132 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2950ea83f60806b49b37bcc6acfd2a2ca16881ae33a99be5d5c9d8e7e62f6555" Nov 01 06:50:32.848313 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:32.848283 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9856327a-ll857" Nov 01 06:50:32.922963 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:32.922926 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f926ff2a-42d9-4af9-a597-c8734d9e1b06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f926ff2a-42d9-4af9-a597-c8734d9e1b06-maas-api-xks-ca\") pod \"f926ff2a-42d9-4af9-a597-c8734d9e1b06\" (UID: \"f926ff2a-42d9-4af9-a597-c8734d9e1b06\") " Nov 01 06:50:32.923138 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:32.923013 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f926ff2a-42d9-4af9-a597-c8734d9e1b06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f926ff2a-42d9-4af9-a597-c8734d9e1b06-maas-api-openshift-ca\") pod \"f926ff2a-42d9-4af9-a597-c8734d9e1b06\" (UID: \"f926ff2a-42d9-4af9-a597-c8734d9e1b06\") " Nov 01 06:50:32.923138 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:32.923067 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f926ff2a-42d9-4af9-a597-c8734d9e1b06-kube-api-access-ndld9\" (UniqueName: \"kubernetes.io/projected/f926ff2a-42d9-4af9-a597-c8734d9e1b06-kube-api-access-ndld9\") pod \"f926ff2a-42d9-4af9-a597-c8734d9e1b06\" (UID: \"f926ff2a-42d9-4af9-a597-c8734d9e1b06\") " Nov 01 06:50:32.923389 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:32.923361 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f926ff2a-42d9-4af9-a597-c8734d9e1b06-maas-api-openshift-ca" pod "f926ff2a-42d9-4af9-a597-c8734d9e1b06" (UID: "f926ff2a-42d9-4af9-a597-c8734d9e1b06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:50:32.923467 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:32.923449 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f926ff2a-42d9-4af9-a597-c8734d9e1b06-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:32.925208 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:32.925187 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f926ff2a-42d9-4af9-a597-c8734d9e1b06-maas-api-xks-ca" pod "f926ff2a-42d9-4af9-a597-c8734d9e1b06" (UID: "f926ff2a-42d9-4af9-a597-c8734d9e1b06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:50:32.925291 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:32.925217 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f926ff2a-42d9-4af9-a597-c8734d9e1b06-kube-api-access-ndld9" pod "f926ff2a-42d9-4af9-a597-c8734d9e1b06" (UID: "f926ff2a-42d9-4af9-a597-c8734d9e1b06"). InnerVolumeSpecName "kube-api-access-ndld9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:50:33.024610 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:33.024569 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndld9\" (UniqueName: \"kubernetes.io/projected/f926ff2a-42d9-4af9-a597-c8734d9e1b06-kube-api-access-ndld9\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:33.024610 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:33.024603 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f926ff2a-42d9-4af9-a597-c8734d9e1b06-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:33.718278 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:33.718242 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9856327a-ll857" Nov 01 06:50:33.718788 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:33.718278 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9856327a-ll857" event={"ID":"f926ff2a-42d9-4af9-a597-c8734d9e1b06","Type":"ContainerDied","Data":"dde61cb41215f249b21df038ad0b68a02ebd3179104fc298e6e8f33f1d309d93"} Nov 01 06:50:33.718788 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:33.718326 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dde61cb41215f249b21df038ad0b68a02ebd3179104fc298e6e8f33f1d309d93" Nov 01 06:50:39.972872 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:50:39.972833 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:50:40.580800 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:40.580765 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b6dc894-pnghr"] Nov 01 06:50:40.581197 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:40.581184 5040 cpu_manager.go:414] "containerMap: removing container" podUID="f926ff2a-42d9-4af9-a597-c8734d9e1b06" containerName="revoke-keys" Nov 01 06:50:40.581247 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:40.581198 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f926ff2a-42d9-4af9-a597-c8734d9e1b06" containerName="revoke-keys" Nov 01 06:50:40.581247 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:40.581234 5040 cpu_manager.go:414] "containerMap: removing container" podUID="3460d306-df7a-4fcd-93ba-0bb4977553e5" containerName="revoke-keys" Nov 01 06:50:40.581247 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:40.581240 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3460d306-df7a-4fcd-93ba-0bb4977553e5" containerName="revoke-keys" Nov 01 06:50:40.581354 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:40.581309 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="3460d306-df7a-4fcd-93ba-0bb4977553e5" containerName="revoke-keys" Nov 01 06:50:40.581354 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:40.581321 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="f926ff2a-42d9-4af9-a597-c8734d9e1b06" containerName="revoke-keys" Nov 01 06:50:40.584738 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:40.584719 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6dc894-pnghr" Nov 01 06:50:40.641730 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:40.641689 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b6dc894-pnghr"] Nov 01 06:50:40.688120 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:40.688078 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:50:40.688304 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:40.688174 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:50:40.688304 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:40.688212 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:50:40.688304 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:40.688230 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:50:40.702996 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:40.702961 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07ee4cb6-922b-4fdd-b2b1-055c5302298a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b6dc894-pnghr\" (UID: \"07ee4cb6-922b-4fdd-b2b1-055c5302298a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6dc894-pnghr" Nov 01 06:50:40.703166 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:40.703093 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07ee4cb6-922b-4fdd-b2b1-055c5302298a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b6dc894-pnghr\" (UID: \"07ee4cb6-922b-4fdd-b2b1-055c5302298a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6dc894-pnghr" Nov 01 06:50:40.703166 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:40.703126 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8gkn\" (UniqueName: \"kubernetes.io/projected/07ee4cb6-922b-4fdd-b2b1-055c5302298a-kube-api-access-s8gkn\") pod \"maas-api-revoke-sub-9b6dc894-pnghr\" (UID: \"07ee4cb6-922b-4fdd-b2b1-055c5302298a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6dc894-pnghr" Nov 01 06:50:40.803972 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:40.803937 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07ee4cb6-922b-4fdd-b2b1-055c5302298a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b6dc894-pnghr\" (UID: \"07ee4cb6-922b-4fdd-b2b1-055c5302298a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6dc894-pnghr" Nov 01 06:50:40.804162 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:40.803981 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8gkn\" (UniqueName: \"kubernetes.io/projected/07ee4cb6-922b-4fdd-b2b1-055c5302298a-kube-api-access-s8gkn\") pod \"maas-api-revoke-sub-9b6dc894-pnghr\" (UID: \"07ee4cb6-922b-4fdd-b2b1-055c5302298a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6dc894-pnghr" Nov 01 06:50:40.804162 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:40.804041 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07ee4cb6-922b-4fdd-b2b1-055c5302298a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b6dc894-pnghr\" (UID: \"07ee4cb6-922b-4fdd-b2b1-055c5302298a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6dc894-pnghr" Nov 01 06:50:40.804776 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:40.804748 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07ee4cb6-922b-4fdd-b2b1-055c5302298a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b6dc894-pnghr\" (UID: \"07ee4cb6-922b-4fdd-b2b1-055c5302298a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6dc894-pnghr" Nov 01 06:50:40.806345 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:40.806319 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07ee4cb6-922b-4fdd-b2b1-055c5302298a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b6dc894-pnghr\" (UID: \"07ee4cb6-922b-4fdd-b2b1-055c5302298a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6dc894-pnghr" Nov 01 06:50:40.866369 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:40.866297 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8gkn\" (UniqueName: \"kubernetes.io/projected/07ee4cb6-922b-4fdd-b2b1-055c5302298a-kube-api-access-s8gkn\") pod \"maas-api-revoke-sub-9b6dc894-pnghr\" (UID: \"07ee4cb6-922b-4fdd-b2b1-055c5302298a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6dc894-pnghr" Nov 01 06:50:40.898067 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:40.898034 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6dc894-pnghr" Nov 01 06:50:41.274328 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:41.274295 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b6dc894-pnghr"] Nov 01 06:50:41.275751 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:50:41.275724 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07ee4cb6_922b_4fdd_b2b1_055c5302298a.slice/crio-21ff4b62584afafb3e4f276e0ea368d3e41ca14fb530c84d7031e8a323997bb2 WatchSource:0}: Error finding container 21ff4b62584afafb3e4f276e0ea368d3e41ca14fb530c84d7031e8a323997bb2: Status 404 returned error can't find the container with id 21ff4b62584afafb3e4f276e0ea368d3e41ca14fb530c84d7031e8a323997bb2 Nov 01 06:50:41.769753 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:41.769714 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6dc894-pnghr" event={"ID":"07ee4cb6-922b-4fdd-b2b1-055c5302298a","Type":"ContainerStarted","Data":"77b73c7a19753cd1fba3d4ef4e4938098aaecbc2a1ab700ccf2eecf3873c91ad"} Nov 01 06:50:41.769929 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:41.769780 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6dc894-pnghr" event={"ID":"07ee4cb6-922b-4fdd-b2b1-055c5302298a","Type":"ContainerStarted","Data":"21ff4b62584afafb3e4f276e0ea368d3e41ca14fb530c84d7031e8a323997bb2"} Nov 01 06:50:41.890856 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:41.890789 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6dc894-pnghr" podStartSLOduration=1.8907740149999999 podStartE2EDuration="1.890774015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:50:41.887793824 +0000 UTC m=+3920.542910992" watchObservedRunningTime="2026-11-01 06:50:41.890774015 +0000 UTC m=+3920.545891181" Nov 01 06:50:42.344580 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:42.344542 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42ca33dc-wtmdf"] Nov 01 06:50:42.348371 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:42.348350 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ca33dc-wtmdf" Nov 01 06:50:42.419535 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:42.418989 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42ca33dc-wtmdf"] Nov 01 06:50:42.432366 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:42.432311 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh2f2\" (UniqueName: \"kubernetes.io/projected/464abd98-219f-4c01-a48c-7572b93f2cf6-kube-api-access-mh2f2\") pod \"maas-api-revoke-sub-42ca33dc-wtmdf\" (UID: \"464abd98-219f-4c01-a48c-7572b93f2cf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ca33dc-wtmdf" Nov 01 06:50:42.432567 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:42.432470 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/464abd98-219f-4c01-a48c-7572b93f2cf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42ca33dc-wtmdf\" (UID: \"464abd98-219f-4c01-a48c-7572b93f2cf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ca33dc-wtmdf" Nov 01 06:50:42.432567 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:42.432544 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/464abd98-219f-4c01-a48c-7572b93f2cf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42ca33dc-wtmdf\" (UID: \"464abd98-219f-4c01-a48c-7572b93f2cf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ca33dc-wtmdf" Nov 01 06:50:42.533612 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:42.533577 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh2f2\" (UniqueName: \"kubernetes.io/projected/464abd98-219f-4c01-a48c-7572b93f2cf6-kube-api-access-mh2f2\") pod \"maas-api-revoke-sub-42ca33dc-wtmdf\" (UID: \"464abd98-219f-4c01-a48c-7572b93f2cf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ca33dc-wtmdf" Nov 01 06:50:42.533819 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:42.533633 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/464abd98-219f-4c01-a48c-7572b93f2cf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42ca33dc-wtmdf\" (UID: \"464abd98-219f-4c01-a48c-7572b93f2cf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ca33dc-wtmdf" Nov 01 06:50:42.533819 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:42.533753 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/464abd98-219f-4c01-a48c-7572b93f2cf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42ca33dc-wtmdf\" (UID: \"464abd98-219f-4c01-a48c-7572b93f2cf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ca33dc-wtmdf" Nov 01 06:50:42.534225 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:42.534197 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/464abd98-219f-4c01-a48c-7572b93f2cf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42ca33dc-wtmdf\" (UID: \"464abd98-219f-4c01-a48c-7572b93f2cf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ca33dc-wtmdf" Nov 01 06:50:42.536054 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:42.536035 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/464abd98-219f-4c01-a48c-7572b93f2cf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42ca33dc-wtmdf\" (UID: \"464abd98-219f-4c01-a48c-7572b93f2cf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ca33dc-wtmdf" Nov 01 06:50:42.595591 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:42.595512 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh2f2\" (UniqueName: \"kubernetes.io/projected/464abd98-219f-4c01-a48c-7572b93f2cf6-kube-api-access-mh2f2\") pod \"maas-api-revoke-sub-42ca33dc-wtmdf\" (UID: \"464abd98-219f-4c01-a48c-7572b93f2cf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ca33dc-wtmdf" Nov 01 06:50:42.661789 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:42.661749 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ca33dc-wtmdf" Nov 01 06:50:42.779248 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:42.779216 5040 generic.go:356] "Generic (PLEG): container finished" podID="07ee4cb6-922b-4fdd-b2b1-055c5302298a" containerID="77b73c7a19753cd1fba3d4ef4e4938098aaecbc2a1ab700ccf2eecf3873c91ad" exitCode=0 Nov 01 06:50:42.779443 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:42.779298 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6dc894-pnghr" event={"ID":"07ee4cb6-922b-4fdd-b2b1-055c5302298a","Type":"ContainerDied","Data":"77b73c7a19753cd1fba3d4ef4e4938098aaecbc2a1ab700ccf2eecf3873c91ad"} Nov 01 06:50:43.042814 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:43.042782 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42ca33dc-wtmdf"] Nov 01 06:50:43.043688 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:50:43.043660 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod464abd98_219f_4c01_a48c_7572b93f2cf6.slice/crio-183a1f061a606b9ef4abde5891a534666bdcc894fe07edb607b3f7ee70e75fda WatchSource:0}: Error finding container 183a1f061a606b9ef4abde5891a534666bdcc894fe07edb607b3f7ee70e75fda: Status 404 returned error can't find the container with id 183a1f061a606b9ef4abde5891a534666bdcc894fe07edb607b3f7ee70e75fda Nov 01 06:50:43.787384 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:43.787344 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ca33dc-wtmdf" event={"ID":"464abd98-219f-4c01-a48c-7572b93f2cf6","Type":"ContainerStarted","Data":"de43637c2774a1a8cd45c84f2f296135b05ab0928fdf8d45e88a71e3a3769e60"} Nov 01 06:50:43.787915 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:43.787408 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ca33dc-wtmdf" event={"ID":"464abd98-219f-4c01-a48c-7572b93f2cf6","Type":"ContainerStarted","Data":"183a1f061a606b9ef4abde5891a534666bdcc894fe07edb607b3f7ee70e75fda"} Nov 01 06:50:43.906829 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:43.906775 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ca33dc-wtmdf" podStartSLOduration=1.906756148 podStartE2EDuration="1.906756148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:50:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:50:43.903797201 +0000 UTC m=+3922.558914367" watchObservedRunningTime="2026-11-01 06:50:43.906756148 +0000 UTC m=+3922.561873329" Nov 01 06:50:43.946649 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:43.946624 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6dc894-pnghr" Nov 01 06:50:44.051189 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:44.051099 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07ee4cb6-922b-4fdd-b2b1-055c5302298a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07ee4cb6-922b-4fdd-b2b1-055c5302298a-maas-api-xks-ca\") pod \"07ee4cb6-922b-4fdd-b2b1-055c5302298a\" (UID: \"07ee4cb6-922b-4fdd-b2b1-055c5302298a\") " Nov 01 06:50:44.051189 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:44.051156 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07ee4cb6-922b-4fdd-b2b1-055c5302298a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07ee4cb6-922b-4fdd-b2b1-055c5302298a-maas-api-openshift-ca\") pod \"07ee4cb6-922b-4fdd-b2b1-055c5302298a\" (UID: \"07ee4cb6-922b-4fdd-b2b1-055c5302298a\") " Nov 01 06:50:44.051440 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:44.051229 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07ee4cb6-922b-4fdd-b2b1-055c5302298a-kube-api-access-s8gkn\" (UniqueName: \"kubernetes.io/projected/07ee4cb6-922b-4fdd-b2b1-055c5302298a-kube-api-access-s8gkn\") pod \"07ee4cb6-922b-4fdd-b2b1-055c5302298a\" (UID: \"07ee4cb6-922b-4fdd-b2b1-055c5302298a\") " Nov 01 06:50:44.051666 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:44.051638 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07ee4cb6-922b-4fdd-b2b1-055c5302298a-maas-api-openshift-ca" pod "07ee4cb6-922b-4fdd-b2b1-055c5302298a" (UID: "07ee4cb6-922b-4fdd-b2b1-055c5302298a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:50:44.053353 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:44.053328 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07ee4cb6-922b-4fdd-b2b1-055c5302298a-kube-api-access-s8gkn" pod "07ee4cb6-922b-4fdd-b2b1-055c5302298a" (UID: "07ee4cb6-922b-4fdd-b2b1-055c5302298a"). InnerVolumeSpecName "kube-api-access-s8gkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:50:44.053458 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:44.053397 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07ee4cb6-922b-4fdd-b2b1-055c5302298a-maas-api-xks-ca" pod "07ee4cb6-922b-4fdd-b2b1-055c5302298a" (UID: "07ee4cb6-922b-4fdd-b2b1-055c5302298a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:50:44.152653 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:44.152616 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07ee4cb6-922b-4fdd-b2b1-055c5302298a-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:44.152653 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:44.152650 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07ee4cb6-922b-4fdd-b2b1-055c5302298a-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:44.152653 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:44.152661 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8gkn\" (UniqueName: \"kubernetes.io/projected/07ee4cb6-922b-4fdd-b2b1-055c5302298a-kube-api-access-s8gkn\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:44.796176 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:44.796141 5040 generic.go:356] "Generic (PLEG): container finished" podID="464abd98-219f-4c01-a48c-7572b93f2cf6" containerID="de43637c2774a1a8cd45c84f2f296135b05ab0928fdf8d45e88a71e3a3769e60" exitCode=0 Nov 01 06:50:44.796662 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:44.796229 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ca33dc-wtmdf" event={"ID":"464abd98-219f-4c01-a48c-7572b93f2cf6","Type":"ContainerDied","Data":"de43637c2774a1a8cd45c84f2f296135b05ab0928fdf8d45e88a71e3a3769e60"} Nov 01 06:50:44.797597 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:44.797567 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6dc894-pnghr" event={"ID":"07ee4cb6-922b-4fdd-b2b1-055c5302298a","Type":"ContainerDied","Data":"21ff4b62584afafb3e4f276e0ea368d3e41ca14fb530c84d7031e8a323997bb2"} Nov 01 06:50:44.797728 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:44.797615 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21ff4b62584afafb3e4f276e0ea368d3e41ca14fb530c84d7031e8a323997bb2" Nov 01 06:50:44.797728 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:44.797579 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6dc894-pnghr" Nov 01 06:50:45.948152 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:45.948124 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ca33dc-wtmdf" Nov 01 06:50:46.072610 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:46.072529 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/464abd98-219f-4c01-a48c-7572b93f2cf6-kube-api-access-mh2f2\" (UniqueName: \"kubernetes.io/projected/464abd98-219f-4c01-a48c-7572b93f2cf6-kube-api-access-mh2f2\") pod \"464abd98-219f-4c01-a48c-7572b93f2cf6\" (UID: \"464abd98-219f-4c01-a48c-7572b93f2cf6\") " Nov 01 06:50:46.072786 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:46.072660 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/464abd98-219f-4c01-a48c-7572b93f2cf6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/464abd98-219f-4c01-a48c-7572b93f2cf6-maas-api-openshift-ca\") pod \"464abd98-219f-4c01-a48c-7572b93f2cf6\" (UID: \"464abd98-219f-4c01-a48c-7572b93f2cf6\") " Nov 01 06:50:46.072786 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:46.072701 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/464abd98-219f-4c01-a48c-7572b93f2cf6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/464abd98-219f-4c01-a48c-7572b93f2cf6-maas-api-xks-ca\") pod \"464abd98-219f-4c01-a48c-7572b93f2cf6\" (UID: \"464abd98-219f-4c01-a48c-7572b93f2cf6\") " Nov 01 06:50:46.073111 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:46.073075 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/464abd98-219f-4c01-a48c-7572b93f2cf6-maas-api-openshift-ca" pod "464abd98-219f-4c01-a48c-7572b93f2cf6" (UID: "464abd98-219f-4c01-a48c-7572b93f2cf6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:50:46.074972 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:46.074944 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/464abd98-219f-4c01-a48c-7572b93f2cf6-maas-api-xks-ca" pod "464abd98-219f-4c01-a48c-7572b93f2cf6" (UID: "464abd98-219f-4c01-a48c-7572b93f2cf6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:50:46.075102 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:46.075088 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/464abd98-219f-4c01-a48c-7572b93f2cf6-kube-api-access-mh2f2" pod "464abd98-219f-4c01-a48c-7572b93f2cf6" (UID: "464abd98-219f-4c01-a48c-7572b93f2cf6"). InnerVolumeSpecName "kube-api-access-mh2f2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:50:46.173801 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:46.173761 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/464abd98-219f-4c01-a48c-7572b93f2cf6-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:46.173801 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:46.173797 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/464abd98-219f-4c01-a48c-7572b93f2cf6-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:46.174028 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:46.173811 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh2f2\" (UniqueName: \"kubernetes.io/projected/464abd98-219f-4c01-a48c-7572b93f2cf6-kube-api-access-mh2f2\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:46.815246 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:46.815215 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ca33dc-wtmdf" Nov 01 06:50:46.815246 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:46.815226 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ca33dc-wtmdf" event={"ID":"464abd98-219f-4c01-a48c-7572b93f2cf6","Type":"ContainerDied","Data":"183a1f061a606b9ef4abde5891a534666bdcc894fe07edb607b3f7ee70e75fda"} Nov 01 06:50:46.815579 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:46.815280 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="183a1f061a606b9ef4abde5891a534666bdcc894fe07edb607b3f7ee70e75fda" Nov 01 06:50:51.052710 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:51.052669 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a1f9505-v49hm"] Nov 01 06:50:51.103305 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:51.103265 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a1f9505-v49hm"] Nov 01 06:50:51.486571 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:51.486455 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29804d8b-zjzkl"] Nov 01 06:50:51.487196 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:51.487167 5040 cpu_manager.go:414] "containerMap: removing container" podUID="07ee4cb6-922b-4fdd-b2b1-055c5302298a" containerName="revoke-keys" Nov 01 06:50:51.487196 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:51.487186 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07ee4cb6-922b-4fdd-b2b1-055c5302298a" containerName="revoke-keys" Nov 01 06:50:51.487397 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:51.487200 5040 cpu_manager.go:414] "containerMap: removing container" podUID="464abd98-219f-4c01-a48c-7572b93f2cf6" containerName="revoke-keys" Nov 01 06:50:51.487397 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:51.487208 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="464abd98-219f-4c01-a48c-7572b93f2cf6" containerName="revoke-keys" Nov 01 06:50:51.487397 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:51.487315 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="07ee4cb6-922b-4fdd-b2b1-055c5302298a" containerName="revoke-keys" Nov 01 06:50:51.487397 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:51.487327 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="464abd98-219f-4c01-a48c-7572b93f2cf6" containerName="revoke-keys" Nov 01 06:50:51.491089 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:51.491064 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29804d8b-zjzkl" Nov 01 06:50:51.516557 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:51.516523 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4324605c-1796-42fa-adeb-b6f401f43645" path="/var/lib/kubelet/pods/4324605c-1796-42fa-adeb-b6f401f43645/volumes" Nov 01 06:50:51.529509 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:51.529464 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3381c8ac-63b7-4df0-b3f5-88579e89489e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29804d8b-zjzkl\" (UID: \"3381c8ac-63b7-4df0-b3f5-88579e89489e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29804d8b-zjzkl" Nov 01 06:50:51.529662 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:51.529599 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r5z8\" (UniqueName: \"kubernetes.io/projected/3381c8ac-63b7-4df0-b3f5-88579e89489e-kube-api-access-6r5z8\") pod \"maas-api-revoke-sub-29804d8b-zjzkl\" (UID: \"3381c8ac-63b7-4df0-b3f5-88579e89489e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29804d8b-zjzkl" Nov 01 06:50:51.529820 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:51.529804 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3381c8ac-63b7-4df0-b3f5-88579e89489e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29804d8b-zjzkl\" (UID: \"3381c8ac-63b7-4df0-b3f5-88579e89489e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29804d8b-zjzkl" Nov 01 06:50:51.544475 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:51.544440 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:50:51.545982 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:51.545952 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29804d8b-zjzkl"] Nov 01 06:50:51.592630 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:51.592592 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:50:51.592630 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:51.592624 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:50:51.592858 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:51.592661 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 06:50:51.631042 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:51.631001 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3381c8ac-63b7-4df0-b3f5-88579e89489e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29804d8b-zjzkl\" (UID: \"3381c8ac-63b7-4df0-b3f5-88579e89489e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29804d8b-zjzkl" Nov 01 06:50:51.631230 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:51.631057 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r5z8\" (UniqueName: \"kubernetes.io/projected/3381c8ac-63b7-4df0-b3f5-88579e89489e-kube-api-access-6r5z8\") pod \"maas-api-revoke-sub-29804d8b-zjzkl\" (UID: \"3381c8ac-63b7-4df0-b3f5-88579e89489e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29804d8b-zjzkl" Nov 01 06:50:51.631230 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:51.631091 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3381c8ac-63b7-4df0-b3f5-88579e89489e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29804d8b-zjzkl\" (UID: \"3381c8ac-63b7-4df0-b3f5-88579e89489e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29804d8b-zjzkl" Nov 01 06:50:51.631883 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:51.631855 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3381c8ac-63b7-4df0-b3f5-88579e89489e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29804d8b-zjzkl\" (UID: \"3381c8ac-63b7-4df0-b3f5-88579e89489e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29804d8b-zjzkl" Nov 01 06:50:51.633550 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:51.633530 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3381c8ac-63b7-4df0-b3f5-88579e89489e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29804d8b-zjzkl\" (UID: \"3381c8ac-63b7-4df0-b3f5-88579e89489e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29804d8b-zjzkl" Nov 01 06:50:51.692832 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:51.692802 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r5z8\" (UniqueName: \"kubernetes.io/projected/3381c8ac-63b7-4df0-b3f5-88579e89489e-kube-api-access-6r5z8\") pod \"maas-api-revoke-sub-29804d8b-zjzkl\" (UID: \"3381c8ac-63b7-4df0-b3f5-88579e89489e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29804d8b-zjzkl" Nov 01 06:50:51.805280 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:51.805240 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29804d8b-zjzkl" Nov 01 06:50:52.183929 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:52.183900 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29804d8b-zjzkl"] Nov 01 06:50:52.184286 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:50:52.184095 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3381c8ac_63b7_4df0_b3f5_88579e89489e.slice/crio-31b080e8007d702c10fb1487311cbb2b5de279d77dc665a30ff7f82bffb804a5 WatchSource:0}: Error finding container 31b080e8007d702c10fb1487311cbb2b5de279d77dc665a30ff7f82bffb804a5: Status 404 returned error can't find the container with id 31b080e8007d702c10fb1487311cbb2b5de279d77dc665a30ff7f82bffb804a5 Nov 01 06:50:52.855051 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:52.854958 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29804d8b-zjzkl" event={"ID":"3381c8ac-63b7-4df0-b3f5-88579e89489e","Type":"ContainerStarted","Data":"6cef364a1fa909d3c7a57e861d028cc604510aa6cb54e2450322f8c1300a0b6e"} Nov 01 06:50:52.855051 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:52.855021 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29804d8b-zjzkl" event={"ID":"3381c8ac-63b7-4df0-b3f5-88579e89489e","Type":"ContainerStarted","Data":"31b080e8007d702c10fb1487311cbb2b5de279d77dc665a30ff7f82bffb804a5"} Nov 01 06:50:52.987145 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:52.987090 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29804d8b-zjzkl" podStartSLOduration=1.987076394 podStartE2EDuration="1.987076394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:50:52.98395163 +0000 UTC m=+3931.639068796" watchObservedRunningTime="2026-11-01 06:50:52.987076394 +0000 UTC m=+3931.642193561" Nov 01 06:50:53.864551 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:53.864511 5040 generic.go:356] "Generic (PLEG): container finished" podID="3381c8ac-63b7-4df0-b3f5-88579e89489e" containerID="6cef364a1fa909d3c7a57e861d028cc604510aa6cb54e2450322f8c1300a0b6e" exitCode=0 Nov 01 06:50:53.864961 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:53.864640 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29804d8b-zjzkl" event={"ID":"3381c8ac-63b7-4df0-b3f5-88579e89489e","Type":"ContainerDied","Data":"6cef364a1fa909d3c7a57e861d028cc604510aa6cb54e2450322f8c1300a0b6e"} Nov 01 06:50:55.012011 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:55.011987 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29804d8b-zjzkl" Nov 01 06:50:55.067245 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:55.067215 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3381c8ac-63b7-4df0-b3f5-88579e89489e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3381c8ac-63b7-4df0-b3f5-88579e89489e-maas-api-openshift-ca\") pod \"3381c8ac-63b7-4df0-b3f5-88579e89489e\" (UID: \"3381c8ac-63b7-4df0-b3f5-88579e89489e\") " Nov 01 06:50:55.067411 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:55.067285 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3381c8ac-63b7-4df0-b3f5-88579e89489e-kube-api-access-6r5z8\" (UniqueName: \"kubernetes.io/projected/3381c8ac-63b7-4df0-b3f5-88579e89489e-kube-api-access-6r5z8\") pod \"3381c8ac-63b7-4df0-b3f5-88579e89489e\" (UID: \"3381c8ac-63b7-4df0-b3f5-88579e89489e\") " Nov 01 06:50:55.067411 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:55.067309 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3381c8ac-63b7-4df0-b3f5-88579e89489e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3381c8ac-63b7-4df0-b3f5-88579e89489e-maas-api-xks-ca\") pod \"3381c8ac-63b7-4df0-b3f5-88579e89489e\" (UID: \"3381c8ac-63b7-4df0-b3f5-88579e89489e\") " Nov 01 06:50:55.067661 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:55.067631 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3381c8ac-63b7-4df0-b3f5-88579e89489e-maas-api-openshift-ca" pod "3381c8ac-63b7-4df0-b3f5-88579e89489e" (UID: "3381c8ac-63b7-4df0-b3f5-88579e89489e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:50:55.069463 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:55.069436 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3381c8ac-63b7-4df0-b3f5-88579e89489e-maas-api-xks-ca" pod "3381c8ac-63b7-4df0-b3f5-88579e89489e" (UID: "3381c8ac-63b7-4df0-b3f5-88579e89489e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:50:55.069574 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:55.069550 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3381c8ac-63b7-4df0-b3f5-88579e89489e-kube-api-access-6r5z8" pod "3381c8ac-63b7-4df0-b3f5-88579e89489e" (UID: "3381c8ac-63b7-4df0-b3f5-88579e89489e"). InnerVolumeSpecName "kube-api-access-6r5z8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:50:55.169205 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:55.169129 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3381c8ac-63b7-4df0-b3f5-88579e89489e-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:55.169205 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:55.169161 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r5z8\" (UniqueName: \"kubernetes.io/projected/3381c8ac-63b7-4df0-b3f5-88579e89489e-kube-api-access-6r5z8\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:55.169205 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:55.169172 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3381c8ac-63b7-4df0-b3f5-88579e89489e-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:55.879212 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:55.879123 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29804d8b-zjzkl" Nov 01 06:50:55.879212 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:55.879161 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29804d8b-zjzkl" event={"ID":"3381c8ac-63b7-4df0-b3f5-88579e89489e","Type":"ContainerDied","Data":"31b080e8007d702c10fb1487311cbb2b5de279d77dc665a30ff7f82bffb804a5"} Nov 01 06:50:55.879212 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:50:55.879209 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31b080e8007d702c10fb1487311cbb2b5de279d77dc665a30ff7f82bffb804a5" Nov 01 06:51:03.431831 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:03.431793 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v"] Nov 01 06:51:04.903396 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:04.903357 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56"] Nov 01 06:51:08.048454 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:08.048379 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09e9f1d9-wwzjb"] Nov 01 06:51:08.098649 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:08.098612 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09e9f1d9-wwzjb"] Nov 01 06:51:09.514795 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:09.514761 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d2a1aa7-fd02-49dd-957b-8659f4299990" path="/var/lib/kubelet/pods/2d2a1aa7-fd02-49dd-957b-8659f4299990/volumes" Nov 01 06:51:09.981841 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:51:09.981745 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:51:18.451829 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:18.451773 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v" podUID="bfa4234e-3def-4d10-b181-c675e28e1b36" containerName="main" containerID="cri-o://6e2991c7efe989dc8c919885bb3d870579049ad0272850eb73a49c3d6a2f06fb" gracePeriod=45 Nov 01 06:51:18.716212 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:18.716145 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v" Nov 01 06:51:18.791214 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:18.791181 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfa4234e-3def-4d10-b181-c675e28e1b36-home\" (UniqueName: \"kubernetes.io/empty-dir/bfa4234e-3def-4d10-b181-c675e28e1b36-home\") pod \"bfa4234e-3def-4d10-b181-c675e28e1b36\" (UID: \"bfa4234e-3def-4d10-b181-c675e28e1b36\") " Nov 01 06:51:18.791403 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:18.791246 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfa4234e-3def-4d10-b181-c675e28e1b36-kube-api-access-lg724\" (UniqueName: \"kubernetes.io/projected/bfa4234e-3def-4d10-b181-c675e28e1b36-kube-api-access-lg724\") pod \"bfa4234e-3def-4d10-b181-c675e28e1b36\" (UID: \"bfa4234e-3def-4d10-b181-c675e28e1b36\") " Nov 01 06:51:18.791403 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:18.791278 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfa4234e-3def-4d10-b181-c675e28e1b36-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfa4234e-3def-4d10-b181-c675e28e1b36-dshm\") pod \"bfa4234e-3def-4d10-b181-c675e28e1b36\" (UID: \"bfa4234e-3def-4d10-b181-c675e28e1b36\") " Nov 01 06:51:18.791403 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:18.791395 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfa4234e-3def-4d10-b181-c675e28e1b36-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfa4234e-3def-4d10-b181-c675e28e1b36-tmp-dir\") pod \"bfa4234e-3def-4d10-b181-c675e28e1b36\" (UID: \"bfa4234e-3def-4d10-b181-c675e28e1b36\") " Nov 01 06:51:18.791613 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:18.791464 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfa4234e-3def-4d10-b181-c675e28e1b36-tls-certs\" (UniqueName: \"kubernetes.io/secret/bfa4234e-3def-4d10-b181-c675e28e1b36-tls-certs\") pod \"bfa4234e-3def-4d10-b181-c675e28e1b36\" (UID: \"bfa4234e-3def-4d10-b181-c675e28e1b36\") " Nov 01 06:51:18.791613 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:18.791526 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfa4234e-3def-4d10-b181-c675e28e1b36-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfa4234e-3def-4d10-b181-c675e28e1b36-model-cache\") pod \"bfa4234e-3def-4d10-b181-c675e28e1b36\" (UID: \"bfa4234e-3def-4d10-b181-c675e28e1b36\") " Nov 01 06:51:18.791613 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:18.791533 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa4234e-3def-4d10-b181-c675e28e1b36-home" pod "bfa4234e-3def-4d10-b181-c675e28e1b36" (UID: "bfa4234e-3def-4d10-b181-c675e28e1b36"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:51:18.791763 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:18.791696 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa4234e-3def-4d10-b181-c675e28e1b36-tmp-dir" pod "bfa4234e-3def-4d10-b181-c675e28e1b36" (UID: "bfa4234e-3def-4d10-b181-c675e28e1b36"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:51:18.791893 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:18.791856 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa4234e-3def-4d10-b181-c675e28e1b36-model-cache" pod "bfa4234e-3def-4d10-b181-c675e28e1b36" (UID: "bfa4234e-3def-4d10-b181-c675e28e1b36"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:51:18.791893 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:18.791888 5040 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfa4234e-3def-4d10-b181-c675e28e1b36-home\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:18.792101 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:18.791906 5040 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfa4234e-3def-4d10-b181-c675e28e1b36-tmp-dir\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:18.793632 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:18.793611 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa4234e-3def-4d10-b181-c675e28e1b36-dshm" pod "bfa4234e-3def-4d10-b181-c675e28e1b36" (UID: "bfa4234e-3def-4d10-b181-c675e28e1b36"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:51:18.793735 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:18.793655 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfa4234e-3def-4d10-b181-c675e28e1b36-tls-certs" pod "bfa4234e-3def-4d10-b181-c675e28e1b36" (UID: "bfa4234e-3def-4d10-b181-c675e28e1b36"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:51:18.793735 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:18.793705 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfa4234e-3def-4d10-b181-c675e28e1b36-kube-api-access-lg724" pod "bfa4234e-3def-4d10-b181-c675e28e1b36" (UID: "bfa4234e-3def-4d10-b181-c675e28e1b36"). InnerVolumeSpecName "kube-api-access-lg724". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:51:18.892711 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:18.892678 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg724\" (UniqueName: \"kubernetes.io/projected/bfa4234e-3def-4d10-b181-c675e28e1b36-kube-api-access-lg724\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:18.892711 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:18.892706 5040 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfa4234e-3def-4d10-b181-c675e28e1b36-dshm\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:18.892711 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:18.892717 5040 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfa4234e-3def-4d10-b181-c675e28e1b36-tls-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:18.892958 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:18.892726 5040 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfa4234e-3def-4d10-b181-c675e28e1b36-model-cache\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:19.031166 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:19.031124 5040 generic.go:356] "Generic (PLEG): container finished" podID="bfa4234e-3def-4d10-b181-c675e28e1b36" containerID="6e2991c7efe989dc8c919885bb3d870579049ad0272850eb73a49c3d6a2f06fb" exitCode=0 Nov 01 06:51:19.031364 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:19.031172 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v" Nov 01 06:51:19.031364 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:19.031213 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v" event={"ID":"bfa4234e-3def-4d10-b181-c675e28e1b36","Type":"ContainerDied","Data":"6e2991c7efe989dc8c919885bb3d870579049ad0272850eb73a49c3d6a2f06fb"} Nov 01 06:51:19.031364 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:19.031259 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v" event={"ID":"bfa4234e-3def-4d10-b181-c675e28e1b36","Type":"ContainerDied","Data":"f17fdd3c6bba2f6a069ca8ff6e78cd47104f1e1e2724652294532261576c2047"} Nov 01 06:51:19.031364 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:19.031284 5040 scope.go:122] "RemoveContainer" containerID="6e2991c7efe989dc8c919885bb3d870579049ad0272850eb73a49c3d6a2f06fb" Nov 01 06:51:19.045293 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:19.045271 5040 scope.go:122] "RemoveContainer" containerID="6e2991c7efe989dc8c919885bb3d870579049ad0272850eb73a49c3d6a2f06fb" Nov 01 06:51:19.045739 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:51:19.045719 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e2991c7efe989dc8c919885bb3d870579049ad0272850eb73a49c3d6a2f06fb\": container with ID starting with 6e2991c7efe989dc8c919885bb3d870579049ad0272850eb73a49c3d6a2f06fb not found: ID does not exist" containerID="6e2991c7efe989dc8c919885bb3d870579049ad0272850eb73a49c3d6a2f06fb" Nov 01 06:51:19.045812 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:19.045746 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e2991c7efe989dc8c919885bb3d870579049ad0272850eb73a49c3d6a2f06fb"} err="failed to get container status \"6e2991c7efe989dc8c919885bb3d870579049ad0272850eb73a49c3d6a2f06fb\": rpc error: code = NotFound desc = could not find container \"6e2991c7efe989dc8c919885bb3d870579049ad0272850eb73a49c3d6a2f06fb\": container with ID starting with 6e2991c7efe989dc8c919885bb3d870579049ad0272850eb73a49c3d6a2f06fb not found: ID does not exist" Nov 01 06:51:19.200179 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:19.200143 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v"] Nov 01 06:51:19.203152 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:19.203129 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-d7d444/sub-test-model-d7d444-kserve-7578986f47-75r8v"] Nov 01 06:51:19.515369 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:19.515337 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfa4234e-3def-4d10-b181-c675e28e1b36" path="/var/lib/kubelet/pods/bfa4234e-3def-4d10-b181-c675e28e1b36/volumes" Nov 01 06:51:19.923065 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:19.922971 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56" podUID="1fbb1642-46b7-4489-bdac-2075d11d644e" containerName="main" containerID="cri-o://1cb2a402c100bf19e79a92b3f0001cca0baabc131df0f1a40e5d33ba717ca9e9" gracePeriod=45 Nov 01 06:51:20.039918 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:20.039885 5040 generic.go:356] "Generic (PLEG): container finished" podID="1fbb1642-46b7-4489-bdac-2075d11d644e" containerID="1cb2a402c100bf19e79a92b3f0001cca0baabc131df0f1a40e5d33ba717ca9e9" exitCode=0 Nov 01 06:51:20.040080 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:20.039951 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56" event={"ID":"1fbb1642-46b7-4489-bdac-2075d11d644e","Type":"ContainerDied","Data":"1cb2a402c100bf19e79a92b3f0001cca0baabc131df0f1a40e5d33ba717ca9e9"} Nov 01 06:51:20.061813 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:20.061778 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56" podUID="1fbb1642-46b7-4489-bdac-2075d11d644e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.230:8000/ready\": dial tcp 10.132.0.230:8000: connect: connection refused" Nov 01 06:51:20.183279 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:20.183225 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56" Nov 01 06:51:20.305336 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:20.305305 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fbb1642-46b7-4489-bdac-2075d11d644e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fbb1642-46b7-4489-bdac-2075d11d644e-model-cache\") pod \"1fbb1642-46b7-4489-bdac-2075d11d644e\" (UID: \"1fbb1642-46b7-4489-bdac-2075d11d644e\") " Nov 01 06:51:20.305524 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:20.305403 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fbb1642-46b7-4489-bdac-2075d11d644e-tls-certs\" (UniqueName: \"kubernetes.io/secret/1fbb1642-46b7-4489-bdac-2075d11d644e-tls-certs\") pod \"1fbb1642-46b7-4489-bdac-2075d11d644e\" (UID: \"1fbb1642-46b7-4489-bdac-2075d11d644e\") " Nov 01 06:51:20.305524 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:20.305431 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fbb1642-46b7-4489-bdac-2075d11d644e-home\" (UniqueName: \"kubernetes.io/empty-dir/1fbb1642-46b7-4489-bdac-2075d11d644e-home\") pod \"1fbb1642-46b7-4489-bdac-2075d11d644e\" (UID: \"1fbb1642-46b7-4489-bdac-2075d11d644e\") " Nov 01 06:51:20.305524 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:20.305470 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fbb1642-46b7-4489-bdac-2075d11d644e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fbb1642-46b7-4489-bdac-2075d11d644e-tmp-dir\") pod \"1fbb1642-46b7-4489-bdac-2075d11d644e\" (UID: \"1fbb1642-46b7-4489-bdac-2075d11d644e\") " Nov 01 06:51:20.305682 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:20.305525 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fbb1642-46b7-4489-bdac-2075d11d644e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fbb1642-46b7-4489-bdac-2075d11d644e-dshm\") pod \"1fbb1642-46b7-4489-bdac-2075d11d644e\" (UID: \"1fbb1642-46b7-4489-bdac-2075d11d644e\") " Nov 01 06:51:20.305682 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:20.305579 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fbb1642-46b7-4489-bdac-2075d11d644e-kube-api-access-wvpb9\" (UniqueName: \"kubernetes.io/projected/1fbb1642-46b7-4489-bdac-2075d11d644e-kube-api-access-wvpb9\") pod \"1fbb1642-46b7-4489-bdac-2075d11d644e\" (UID: \"1fbb1642-46b7-4489-bdac-2075d11d644e\") " Nov 01 06:51:20.305802 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:20.305646 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fbb1642-46b7-4489-bdac-2075d11d644e-model-cache" pod "1fbb1642-46b7-4489-bdac-2075d11d644e" (UID: "1fbb1642-46b7-4489-bdac-2075d11d644e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:51:20.305802 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:20.305721 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fbb1642-46b7-4489-bdac-2075d11d644e-home" pod "1fbb1642-46b7-4489-bdac-2075d11d644e" (UID: "1fbb1642-46b7-4489-bdac-2075d11d644e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:51:20.305922 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:20.305832 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fbb1642-46b7-4489-bdac-2075d11d644e-tmp-dir" pod "1fbb1642-46b7-4489-bdac-2075d11d644e" (UID: "1fbb1642-46b7-4489-bdac-2075d11d644e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:51:20.305980 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:20.305920 5040 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fbb1642-46b7-4489-bdac-2075d11d644e-home\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:20.305980 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:20.305938 5040 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fbb1642-46b7-4489-bdac-2075d11d644e-tmp-dir\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:20.305980 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:20.305952 5040 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fbb1642-46b7-4489-bdac-2075d11d644e-model-cache\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:20.307873 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:20.307848 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fbb1642-46b7-4489-bdac-2075d11d644e-kube-api-access-wvpb9" pod "1fbb1642-46b7-4489-bdac-2075d11d644e" (UID: "1fbb1642-46b7-4489-bdac-2075d11d644e"). InnerVolumeSpecName "kube-api-access-wvpb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:51:20.308025 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:20.307886 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fbb1642-46b7-4489-bdac-2075d11d644e-dshm" pod "1fbb1642-46b7-4489-bdac-2075d11d644e" (UID: "1fbb1642-46b7-4489-bdac-2075d11d644e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:51:20.308143 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:20.308122 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fbb1642-46b7-4489-bdac-2075d11d644e-tls-certs" pod "1fbb1642-46b7-4489-bdac-2075d11d644e" (UID: "1fbb1642-46b7-4489-bdac-2075d11d644e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:51:20.406935 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:20.406891 5040 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fbb1642-46b7-4489-bdac-2075d11d644e-tls-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:20.406935 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:20.406928 5040 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fbb1642-46b7-4489-bdac-2075d11d644e-dshm\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:20.406935 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:20.406938 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvpb9\" (UniqueName: \"kubernetes.io/projected/1fbb1642-46b7-4489-bdac-2075d11d644e-kube-api-access-wvpb9\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:21.048943 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:21.048909 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56" Nov 01 06:51:21.049332 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:21.048913 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56" event={"ID":"1fbb1642-46b7-4489-bdac-2075d11d644e","Type":"ContainerDied","Data":"2759774f16906f1a736ca879d65d960bb332bd45282ae095b3199fff719d24d4"} Nov 01 06:51:21.049332 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:21.049073 5040 scope.go:122] "RemoveContainer" containerID="1cb2a402c100bf19e79a92b3f0001cca0baabc131df0f1a40e5d33ba717ca9e9" Nov 01 06:51:21.216551 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:21.216511 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56"] Nov 01 06:51:21.220117 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:21.220088 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-747b7b/sub-test-model-747b7b-kserve-65f6846b4c-wsx56"] Nov 01 06:51:21.515059 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:21.515031 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fbb1642-46b7-4489-bdac-2075d11d644e" path="/var/lib/kubelet/pods/1fbb1642-46b7-4489-bdac-2075d11d644e/volumes" Nov 01 06:51:24.043684 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:24.043652 5040 scope.go:122] "RemoveContainer" containerID="1acd03eaae2dcd1ebb9264f6e36a483f2c07c834f3e5fdd208c229ee7f72e804" Nov 01 06:51:31.416625 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:31.416588 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq"] Nov 01 06:51:31.417034 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:31.416836 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" podUID="cee73e53-f9e2-49c9-a19c-ee33cc2ac68e" containerName="istio-proxy" containerID="cri-o://560e4e560172830fcd6e0f82fda9e50047d90419514418280660f6941eab990d" gracePeriod=30 Nov 01 06:51:36.684439 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:36.684375 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:51:36.775368 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:36.775330 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-workload-certs\") pod \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\" (UID: \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\") " Nov 01 06:51:36.775600 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:36.775375 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-kube-api-access-mmjpg\" (UniqueName: \"kubernetes.io/projected/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-kube-api-access-mmjpg\") pod \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\" (UID: \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\") " Nov 01 06:51:36.775600 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:36.775417 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-istio-envoy\") pod \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\" (UID: \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\") " Nov 01 06:51:36.775600 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:36.775450 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-workload-socket\") pod \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\" (UID: \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\") " Nov 01 06:51:36.775600 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:36.775502 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-istio-podinfo\") pod \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\" (UID: \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\") " Nov 01 06:51:36.775600 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:36.775593 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-istio-token\" (UniqueName: \"kubernetes.io/projected/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-istio-token\") pod \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\" (UID: \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\") " Nov 01 06:51:36.775898 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:36.775648 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-credential-socket\") pod \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\" (UID: \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\") " Nov 01 06:51:36.775898 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:36.775709 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-istiod-ca-cert\") pod \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\" (UID: \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\") " Nov 01 06:51:36.775898 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:36.775707 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-workload-certs" pod "cee73e53-f9e2-49c9-a19c-ee33cc2ac68e" (UID: "cee73e53-f9e2-49c9-a19c-ee33cc2ac68e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:51:36.775898 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:36.775758 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-istio-data\") pod \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\" (UID: \"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e\") " Nov 01 06:51:36.775898 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:36.775813 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-workload-socket" pod "cee73e53-f9e2-49c9-a19c-ee33cc2ac68e" (UID: "cee73e53-f9e2-49c9-a19c-ee33cc2ac68e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:51:36.776163 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:36.775936 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-credential-socket" pod "cee73e53-f9e2-49c9-a19c-ee33cc2ac68e" (UID: "cee73e53-f9e2-49c9-a19c-ee33cc2ac68e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:51:36.776163 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:36.776057 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-istiod-ca-cert" pod "cee73e53-f9e2-49c9-a19c-ee33cc2ac68e" (UID: "cee73e53-f9e2-49c9-a19c-ee33cc2ac68e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:51:36.776163 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:36.776066 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-istio-data" pod "cee73e53-f9e2-49c9-a19c-ee33cc2ac68e" (UID: "cee73e53-f9e2-49c9-a19c-ee33cc2ac68e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:51:36.776163 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:36.776126 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:36.776163 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:36.776145 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:36.776163 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:36.776155 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:36.776163 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:36.776165 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:36.777995 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:36.777966 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-istio-envoy" pod "cee73e53-f9e2-49c9-a19c-ee33cc2ac68e" (UID: "cee73e53-f9e2-49c9-a19c-ee33cc2ac68e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:51:36.777995 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:36.777978 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-kube-api-access-mmjpg" pod "cee73e53-f9e2-49c9-a19c-ee33cc2ac68e" (UID: "cee73e53-f9e2-49c9-a19c-ee33cc2ac68e"). InnerVolumeSpecName "kube-api-access-mmjpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:51:36.778185 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:36.778147 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-istio-podinfo" pod "cee73e53-f9e2-49c9-a19c-ee33cc2ac68e" (UID: "cee73e53-f9e2-49c9-a19c-ee33cc2ac68e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:51:36.778185 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:36.778158 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-istio-token" pod "cee73e53-f9e2-49c9-a19c-ee33cc2ac68e" (UID: "cee73e53-f9e2-49c9-a19c-ee33cc2ac68e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:51:36.876866 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:36.876832 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmjpg\" (UniqueName: \"kubernetes.io/projected/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-kube-api-access-mmjpg\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:36.876866 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:36.876865 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:36.877071 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:36.876879 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:36.877071 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:36.876892 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:36.877071 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:36.876904 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:37.152128 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:37.152088 5040 generic.go:356] "Generic (PLEG): container finished" podID="cee73e53-f9e2-49c9-a19c-ee33cc2ac68e" containerID="560e4e560172830fcd6e0f82fda9e50047d90419514418280660f6941eab990d" exitCode=0 Nov 01 06:51:37.152334 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:37.152160 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" Nov 01 06:51:37.152334 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:37.152171 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" event={"ID":"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e","Type":"ContainerDied","Data":"560e4e560172830fcd6e0f82fda9e50047d90419514418280660f6941eab990d"} Nov 01 06:51:37.152334 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:37.152263 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq" event={"ID":"cee73e53-f9e2-49c9-a19c-ee33cc2ac68e","Type":"ContainerDied","Data":"757a02f344613430ccf1e3539d1f76ff6d24c3334824c901ebc8e592817eacc4"} Nov 01 06:51:37.152334 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:37.152294 5040 scope.go:122] "RemoveContainer" containerID="560e4e560172830fcd6e0f82fda9e50047d90419514418280660f6941eab990d" Nov 01 06:51:37.162684 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:37.162665 5040 scope.go:122] "RemoveContainer" containerID="1a88b3ba3539e775c2940edeeb343c0a783531829cc0618570eb68cbb26bfd64" Nov 01 06:51:37.171438 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:37.171414 5040 scope.go:122] "RemoveContainer" containerID="560e4e560172830fcd6e0f82fda9e50047d90419514418280660f6941eab990d" Nov 01 06:51:37.171710 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:51:37.171689 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"560e4e560172830fcd6e0f82fda9e50047d90419514418280660f6941eab990d\": container with ID starting with 560e4e560172830fcd6e0f82fda9e50047d90419514418280660f6941eab990d not found: ID does not exist" containerID="560e4e560172830fcd6e0f82fda9e50047d90419514418280660f6941eab990d" Nov 01 06:51:37.171781 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:37.171719 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"560e4e560172830fcd6e0f82fda9e50047d90419514418280660f6941eab990d"} err="failed to get container status \"560e4e560172830fcd6e0f82fda9e50047d90419514418280660f6941eab990d\": rpc error: code = NotFound desc = could not find container \"560e4e560172830fcd6e0f82fda9e50047d90419514418280660f6941eab990d\": container with ID starting with 560e4e560172830fcd6e0f82fda9e50047d90419514418280660f6941eab990d not found: ID does not exist" Nov 01 06:51:37.171781 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:37.171737 5040 scope.go:122] "RemoveContainer" containerID="1a88b3ba3539e775c2940edeeb343c0a783531829cc0618570eb68cbb26bfd64" Nov 01 06:51:37.171988 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:51:37.171968 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a88b3ba3539e775c2940edeeb343c0a783531829cc0618570eb68cbb26bfd64\": container with ID starting with 1a88b3ba3539e775c2940edeeb343c0a783531829cc0618570eb68cbb26bfd64 not found: ID does not exist" containerID="1a88b3ba3539e775c2940edeeb343c0a783531829cc0618570eb68cbb26bfd64" Nov 01 06:51:37.172036 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:37.171995 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a88b3ba3539e775c2940edeeb343c0a783531829cc0618570eb68cbb26bfd64"} err="failed to get container status \"1a88b3ba3539e775c2940edeeb343c0a783531829cc0618570eb68cbb26bfd64\": rpc error: code = NotFound desc = could not find container \"1a88b3ba3539e775c2940edeeb343c0a783531829cc0618570eb68cbb26bfd64\": container with ID starting with 1a88b3ba3539e775c2940edeeb343c0a783531829cc0618570eb68cbb26bfd64 not found: ID does not exist" Nov 01 06:51:37.319667 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:37.319633 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq"] Nov 01 06:51:37.370156 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:37.370119 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-5be4fa-openshift-default-55fd78d55c-9k9mq"] Nov 01 06:51:37.514426 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:51:37.514375 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cee73e53-f9e2-49c9-a19c-ee33cc2ac68e" path="/var/lib/kubelet/pods/cee73e53-f9e2-49c9-a19c-ee33cc2ac68e/volumes" Nov 01 06:51:39.973414 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:51:39.973378 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:52:01.410849 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:01.410757 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp"] Nov 01 06:52:01.411353 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:01.411119 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" podUID="bb482d61-0774-4394-8c37-a564ef5b6838" containerName="istio-proxy" containerID="cri-o://c2d5a44574e65916a9492d03badb46b392fdf88d0df62258518a4f9a0f759890" gracePeriod=30 Nov 01 06:52:06.670225 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:06.670200 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:52:06.735351 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:06.735266 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb482d61-0774-4394-8c37-a564ef5b6838-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb482d61-0774-4394-8c37-a564ef5b6838-workload-socket\") pod \"bb482d61-0774-4394-8c37-a564ef5b6838\" (UID: \"bb482d61-0774-4394-8c37-a564ef5b6838\") " Nov 01 06:52:06.735351 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:06.735305 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb482d61-0774-4394-8c37-a564ef5b6838-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb482d61-0774-4394-8c37-a564ef5b6838-istio-envoy\") pod \"bb482d61-0774-4394-8c37-a564ef5b6838\" (UID: \"bb482d61-0774-4394-8c37-a564ef5b6838\") " Nov 01 06:52:06.735351 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:06.735346 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb482d61-0774-4394-8c37-a564ef5b6838-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb482d61-0774-4394-8c37-a564ef5b6838-workload-certs\") pod \"bb482d61-0774-4394-8c37-a564ef5b6838\" (UID: \"bb482d61-0774-4394-8c37-a564ef5b6838\") " Nov 01 06:52:06.735692 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:06.735369 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb482d61-0774-4394-8c37-a564ef5b6838-istio-token\" (UniqueName: \"kubernetes.io/projected/bb482d61-0774-4394-8c37-a564ef5b6838-istio-token\") pod \"bb482d61-0774-4394-8c37-a564ef5b6838\" (UID: \"bb482d61-0774-4394-8c37-a564ef5b6838\") " Nov 01 06:52:06.735692 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:06.735409 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb482d61-0774-4394-8c37-a564ef5b6838-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb482d61-0774-4394-8c37-a564ef5b6838-credential-socket\") pod \"bb482d61-0774-4394-8c37-a564ef5b6838\" (UID: \"bb482d61-0774-4394-8c37-a564ef5b6838\") " Nov 01 06:52:06.735692 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:06.735434 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb482d61-0774-4394-8c37-a564ef5b6838-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb482d61-0774-4394-8c37-a564ef5b6838-istiod-ca-cert\") pod \"bb482d61-0774-4394-8c37-a564ef5b6838\" (UID: \"bb482d61-0774-4394-8c37-a564ef5b6838\") " Nov 01 06:52:06.735692 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:06.735462 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb482d61-0774-4394-8c37-a564ef5b6838-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb482d61-0774-4394-8c37-a564ef5b6838-istio-podinfo\") pod \"bb482d61-0774-4394-8c37-a564ef5b6838\" (UID: \"bb482d61-0774-4394-8c37-a564ef5b6838\") " Nov 01 06:52:06.735692 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:06.735527 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb482d61-0774-4394-8c37-a564ef5b6838-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb482d61-0774-4394-8c37-a564ef5b6838-istio-data\") pod \"bb482d61-0774-4394-8c37-a564ef5b6838\" (UID: \"bb482d61-0774-4394-8c37-a564ef5b6838\") " Nov 01 06:52:06.735692 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:06.735589 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb482d61-0774-4394-8c37-a564ef5b6838-kube-api-access-k2pwq\" (UniqueName: \"kubernetes.io/projected/bb482d61-0774-4394-8c37-a564ef5b6838-kube-api-access-k2pwq\") pod \"bb482d61-0774-4394-8c37-a564ef5b6838\" (UID: \"bb482d61-0774-4394-8c37-a564ef5b6838\") " Nov 01 06:52:06.735692 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:06.735623 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb482d61-0774-4394-8c37-a564ef5b6838-workload-socket" pod "bb482d61-0774-4394-8c37-a564ef5b6838" (UID: "bb482d61-0774-4394-8c37-a564ef5b6838"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:52:06.736053 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:06.735744 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb482d61-0774-4394-8c37-a564ef5b6838-workload-certs" pod "bb482d61-0774-4394-8c37-a564ef5b6838" (UID: "bb482d61-0774-4394-8c37-a564ef5b6838"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:52:06.736053 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:06.735763 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb482d61-0774-4394-8c37-a564ef5b6838-credential-socket" pod "bb482d61-0774-4394-8c37-a564ef5b6838" (UID: "bb482d61-0774-4394-8c37-a564ef5b6838"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:52:06.736053 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:06.735904 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb482d61-0774-4394-8c37-a564ef5b6838-istio-data" pod "bb482d61-0774-4394-8c37-a564ef5b6838" (UID: "bb482d61-0774-4394-8c37-a564ef5b6838"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:52:06.736053 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:06.736031 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb482d61-0774-4394-8c37-a564ef5b6838-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:52:06.736053 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:06.736046 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb482d61-0774-4394-8c37-a564ef5b6838-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:52:06.736053 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:06.736056 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb482d61-0774-4394-8c37-a564ef5b6838-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:52:06.736401 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:06.736065 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb482d61-0774-4394-8c37-a564ef5b6838-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:52:06.736401 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:06.736224 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb482d61-0774-4394-8c37-a564ef5b6838-istiod-ca-cert" pod "bb482d61-0774-4394-8c37-a564ef5b6838" (UID: "bb482d61-0774-4394-8c37-a564ef5b6838"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:52:06.737692 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:06.737666 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb482d61-0774-4394-8c37-a564ef5b6838-istio-envoy" pod "bb482d61-0774-4394-8c37-a564ef5b6838" (UID: "bb482d61-0774-4394-8c37-a564ef5b6838"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:52:06.737818 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:06.737737 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb482d61-0774-4394-8c37-a564ef5b6838-istio-podinfo" pod "bb482d61-0774-4394-8c37-a564ef5b6838" (UID: "bb482d61-0774-4394-8c37-a564ef5b6838"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:52:06.738021 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:06.738002 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb482d61-0774-4394-8c37-a564ef5b6838-kube-api-access-k2pwq" pod "bb482d61-0774-4394-8c37-a564ef5b6838" (UID: "bb482d61-0774-4394-8c37-a564ef5b6838"). InnerVolumeSpecName "kube-api-access-k2pwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:52:06.738138 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:06.738018 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb482d61-0774-4394-8c37-a564ef5b6838-istio-token" pod "bb482d61-0774-4394-8c37-a564ef5b6838" (UID: "bb482d61-0774-4394-8c37-a564ef5b6838"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:52:06.837439 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:06.837403 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2pwq\" (UniqueName: \"kubernetes.io/projected/bb482d61-0774-4394-8c37-a564ef5b6838-kube-api-access-k2pwq\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:52:06.837439 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:06.837435 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb482d61-0774-4394-8c37-a564ef5b6838-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:52:06.837439 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:06.837444 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb482d61-0774-4394-8c37-a564ef5b6838-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:52:06.837700 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:06.837453 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb482d61-0774-4394-8c37-a564ef5b6838-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:52:06.837700 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:06.837461 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb482d61-0774-4394-8c37-a564ef5b6838-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:52:07.345957 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:07.345916 5040 generic.go:356] "Generic (PLEG): container finished" podID="bb482d61-0774-4394-8c37-a564ef5b6838" containerID="c2d5a44574e65916a9492d03badb46b392fdf88d0df62258518a4f9a0f759890" exitCode=0 Nov 01 06:52:07.346135 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:07.345973 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" Nov 01 06:52:07.346135 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:07.345993 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" event={"ID":"bb482d61-0774-4394-8c37-a564ef5b6838","Type":"ContainerDied","Data":"c2d5a44574e65916a9492d03badb46b392fdf88d0df62258518a4f9a0f759890"} Nov 01 06:52:07.346135 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:07.346051 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp" event={"ID":"bb482d61-0774-4394-8c37-a564ef5b6838","Type":"ContainerDied","Data":"12dc377a1d60b60ce112296d4125edfea9d93846a1432702b4fbcc082927cb34"} Nov 01 06:52:07.346135 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:07.346073 5040 scope.go:122] "RemoveContainer" containerID="c2d5a44574e65916a9492d03badb46b392fdf88d0df62258518a4f9a0f759890" Nov 01 06:52:07.356905 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:07.356886 5040 scope.go:122] "RemoveContainer" containerID="a83bdbd5fe62ee118d2b1f3f325489a615d4b59532599da5a32e0f396fd47809" Nov 01 06:52:07.365880 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:07.365858 5040 scope.go:122] "RemoveContainer" containerID="c2d5a44574e65916a9492d03badb46b392fdf88d0df62258518a4f9a0f759890" Nov 01 06:52:07.366158 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:52:07.366139 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2d5a44574e65916a9492d03badb46b392fdf88d0df62258518a4f9a0f759890\": container with ID starting with c2d5a44574e65916a9492d03badb46b392fdf88d0df62258518a4f9a0f759890 not found: ID does not exist" containerID="c2d5a44574e65916a9492d03badb46b392fdf88d0df62258518a4f9a0f759890" Nov 01 06:52:07.366211 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:07.366167 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2d5a44574e65916a9492d03badb46b392fdf88d0df62258518a4f9a0f759890"} err="failed to get container status \"c2d5a44574e65916a9492d03badb46b392fdf88d0df62258518a4f9a0f759890\": rpc error: code = NotFound desc = could not find container \"c2d5a44574e65916a9492d03badb46b392fdf88d0df62258518a4f9a0f759890\": container with ID starting with c2d5a44574e65916a9492d03badb46b392fdf88d0df62258518a4f9a0f759890 not found: ID does not exist" Nov 01 06:52:07.366211 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:07.366184 5040 scope.go:122] "RemoveContainer" containerID="a83bdbd5fe62ee118d2b1f3f325489a615d4b59532599da5a32e0f396fd47809" Nov 01 06:52:07.366461 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:52:07.366445 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a83bdbd5fe62ee118d2b1f3f325489a615d4b59532599da5a32e0f396fd47809\": container with ID starting with a83bdbd5fe62ee118d2b1f3f325489a615d4b59532599da5a32e0f396fd47809 not found: ID does not exist" containerID="a83bdbd5fe62ee118d2b1f3f325489a615d4b59532599da5a32e0f396fd47809" Nov 01 06:52:07.366538 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:07.366463 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a83bdbd5fe62ee118d2b1f3f325489a615d4b59532599da5a32e0f396fd47809"} err="failed to get container status \"a83bdbd5fe62ee118d2b1f3f325489a615d4b59532599da5a32e0f396fd47809\": rpc error: code = NotFound desc = could not find container \"a83bdbd5fe62ee118d2b1f3f325489a615d4b59532599da5a32e0f396fd47809\": container with ID starting with a83bdbd5fe62ee118d2b1f3f325489a615d4b59532599da5a32e0f396fd47809 not found: ID does not exist" Nov 01 06:52:07.515756 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:07.515725 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp"] Nov 01 06:52:07.566611 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:07.566579 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d7d444-openshift-default-758d69977f-f95wp"] Nov 01 06:52:09.514617 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:09.514586 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb482d61-0774-4394-8c37-a564ef5b6838" path="/var/lib/kubelet/pods/bb482d61-0774-4394-8c37-a564ef5b6838/volumes" Nov 01 06:52:09.972635 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:52:09.972553 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:52:24.078781 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:24.078745 5040 scope.go:122] "RemoveContainer" containerID="dfd0a051d16610fe0e76b6834add93c99ddde51a0cd75c75eccb31d1238d7d82" Nov 01 06:52:24.090637 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:24.090608 5040 scope.go:122] "RemoveContainer" containerID="67802d218ad04ddf14dbb05ba0440793aff3438620c89fa036f6205369735508" Nov 01 06:52:34.768727 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:34.768689 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-71cea4-589654b775-j76mx"] Nov 01 06:52:34.868569 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:34.868528 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71cea4-799667f47-zx5bs"] Nov 01 06:52:39.714140 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:39.714102 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq"] Nov 01 06:52:39.714704 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:39.714326 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" podUID="14b4da92-5164-45df-8306-5f228a7c5ac2" containerName="istio-proxy" containerID="cri-o://61e42cad06057a6933769c825bb406fe1448b60b24169e0bcd8a699045509ed9" gracePeriod=30 Nov 01 06:52:39.869233 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:39.869176 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71cea4-799667f47-zx5bs" podUID="232b8e8b-3616-4768-9741-50e93d149e4b" containerName="payload-pre-processing" containerID="cri-o://98ca89d7a9974d487c924ddcc9c9299c19d85897a09a2e9a0d55a08a3acd69b2" gracePeriod=25 Nov 01 06:52:39.972276 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:52:39.972184 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:52:40.514654 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:40.514605 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71cea4-589654b775-j76mx" podUID="785f90ff-95cf-4273-9952-6877d4ffb72d" containerName="payload-processing" containerID="cri-o://c35c43d2b8f46576a8609704f55e86afaae337c347947802b1e8b052934badc1" gracePeriod=25 Nov 01 06:52:41.797268 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:41.797226 5040 patch_prober.go:29] interesting pod/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 06:52:41.797676 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:41.797299 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" podUID="14b4da92-5164-45df-8306-5f228a7c5ac2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 06:52:43.057629 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:43.057595 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5464df48-6fstk"] Nov 01 06:52:43.108134 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:43.108100 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5464df48-6fstk"] Nov 01 06:52:43.514441 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:43.514407 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50aebf01-af0d-4da8-9f2a-daee6a73880b" path="/var/lib/kubelet/pods/50aebf01-af0d-4da8-9f2a-daee6a73880b/volumes" Nov 01 06:52:44.971343 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:44.971314 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:52:44.993775 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:44.993710 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b4da92-5164-45df-8306-5f228a7c5ac2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b4da92-5164-45df-8306-5f228a7c5ac2-workload-socket\") pod \"14b4da92-5164-45df-8306-5f228a7c5ac2\" (UID: \"14b4da92-5164-45df-8306-5f228a7c5ac2\") " Nov 01 06:52:44.993775 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:44.993745 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14b4da92-5164-45df-8306-5f228a7c5ac2-kube-api-access-vgprk\" (UniqueName: \"kubernetes.io/projected/14b4da92-5164-45df-8306-5f228a7c5ac2-kube-api-access-vgprk\") pod \"14b4da92-5164-45df-8306-5f228a7c5ac2\" (UID: \"14b4da92-5164-45df-8306-5f228a7c5ac2\") " Nov 01 06:52:44.993775 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:44.993772 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b4da92-5164-45df-8306-5f228a7c5ac2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14b4da92-5164-45df-8306-5f228a7c5ac2-istio-envoy\") pod \"14b4da92-5164-45df-8306-5f228a7c5ac2\" (UID: \"14b4da92-5164-45df-8306-5f228a7c5ac2\") " Nov 01 06:52:44.993980 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:44.993811 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14b4da92-5164-45df-8306-5f228a7c5ac2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14b4da92-5164-45df-8306-5f228a7c5ac2-istiod-ca-cert\") pod \"14b4da92-5164-45df-8306-5f228a7c5ac2\" (UID: \"14b4da92-5164-45df-8306-5f228a7c5ac2\") " Nov 01 06:52:44.993980 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:44.993840 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14b4da92-5164-45df-8306-5f228a7c5ac2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14b4da92-5164-45df-8306-5f228a7c5ac2-istio-podinfo\") pod \"14b4da92-5164-45df-8306-5f228a7c5ac2\" (UID: \"14b4da92-5164-45df-8306-5f228a7c5ac2\") " Nov 01 06:52:44.993980 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:44.993948 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b4da92-5164-45df-8306-5f228a7c5ac2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b4da92-5164-45df-8306-5f228a7c5ac2-credential-socket\") pod \"14b4da92-5164-45df-8306-5f228a7c5ac2\" (UID: \"14b4da92-5164-45df-8306-5f228a7c5ac2\") " Nov 01 06:52:44.994151 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:44.993988 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b4da92-5164-45df-8306-5f228a7c5ac2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14b4da92-5164-45df-8306-5f228a7c5ac2-istio-data\") pod \"14b4da92-5164-45df-8306-5f228a7c5ac2\" (UID: \"14b4da92-5164-45df-8306-5f228a7c5ac2\") " Nov 01 06:52:44.994151 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:44.994019 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14b4da92-5164-45df-8306-5f228a7c5ac2-istio-token\" (UniqueName: \"kubernetes.io/projected/14b4da92-5164-45df-8306-5f228a7c5ac2-istio-token\") pod \"14b4da92-5164-45df-8306-5f228a7c5ac2\" (UID: \"14b4da92-5164-45df-8306-5f228a7c5ac2\") " Nov 01 06:52:44.994151 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:44.994041 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b4da92-5164-45df-8306-5f228a7c5ac2-workload-socket" pod "14b4da92-5164-45df-8306-5f228a7c5ac2" (UID: "14b4da92-5164-45df-8306-5f228a7c5ac2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:52:44.994151 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:44.994053 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b4da92-5164-45df-8306-5f228a7c5ac2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14b4da92-5164-45df-8306-5f228a7c5ac2-workload-certs\") pod \"14b4da92-5164-45df-8306-5f228a7c5ac2\" (UID: \"14b4da92-5164-45df-8306-5f228a7c5ac2\") " Nov 01 06:52:44.994388 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:44.994256 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b4da92-5164-45df-8306-5f228a7c5ac2-credential-socket" pod "14b4da92-5164-45df-8306-5f228a7c5ac2" (UID: "14b4da92-5164-45df-8306-5f228a7c5ac2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:52:44.994388 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:44.994281 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b4da92-5164-45df-8306-5f228a7c5ac2-workload-certs" pod "14b4da92-5164-45df-8306-5f228a7c5ac2" (UID: "14b4da92-5164-45df-8306-5f228a7c5ac2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:52:44.994388 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:44.994336 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14b4da92-5164-45df-8306-5f228a7c5ac2-istiod-ca-cert" pod "14b4da92-5164-45df-8306-5f228a7c5ac2" (UID: "14b4da92-5164-45df-8306-5f228a7c5ac2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:52:44.994388 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:44.994358 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b4da92-5164-45df-8306-5f228a7c5ac2-istio-data" pod "14b4da92-5164-45df-8306-5f228a7c5ac2" (UID: "14b4da92-5164-45df-8306-5f228a7c5ac2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:52:44.994653 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:44.994467 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b4da92-5164-45df-8306-5f228a7c5ac2-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:52:44.994653 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:44.994502 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14b4da92-5164-45df-8306-5f228a7c5ac2-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:52:44.994653 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:44.994517 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14b4da92-5164-45df-8306-5f228a7c5ac2-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:52:44.994653 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:44.994529 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b4da92-5164-45df-8306-5f228a7c5ac2-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:52:44.994653 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:44.994537 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14b4da92-5164-45df-8306-5f228a7c5ac2-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:52:44.996128 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:44.996101 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14b4da92-5164-45df-8306-5f228a7c5ac2-istio-podinfo" pod "14b4da92-5164-45df-8306-5f228a7c5ac2" (UID: "14b4da92-5164-45df-8306-5f228a7c5ac2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:52:44.996240 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:44.996210 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14b4da92-5164-45df-8306-5f228a7c5ac2-istio-token" pod "14b4da92-5164-45df-8306-5f228a7c5ac2" (UID: "14b4da92-5164-45df-8306-5f228a7c5ac2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:52:44.996442 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:44.996424 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14b4da92-5164-45df-8306-5f228a7c5ac2-kube-api-access-vgprk" pod "14b4da92-5164-45df-8306-5f228a7c5ac2" (UID: "14b4da92-5164-45df-8306-5f228a7c5ac2"). InnerVolumeSpecName "kube-api-access-vgprk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:52:44.996507 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:44.996438 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b4da92-5164-45df-8306-5f228a7c5ac2-istio-envoy" pod "14b4da92-5164-45df-8306-5f228a7c5ac2" (UID: "14b4da92-5164-45df-8306-5f228a7c5ac2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:52:45.095921 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:45.095885 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14b4da92-5164-45df-8306-5f228a7c5ac2-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:52:45.095921 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:45.095916 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgprk\" (UniqueName: \"kubernetes.io/projected/14b4da92-5164-45df-8306-5f228a7c5ac2-kube-api-access-vgprk\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:52:45.095921 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:45.095927 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14b4da92-5164-45df-8306-5f228a7c5ac2-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:52:45.096162 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:45.095936 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14b4da92-5164-45df-8306-5f228a7c5ac2-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:52:45.574793 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:45.574760 5040 generic.go:356] "Generic (PLEG): container finished" podID="14b4da92-5164-45df-8306-5f228a7c5ac2" containerID="61e42cad06057a6933769c825bb406fe1448b60b24169e0bcd8a699045509ed9" exitCode=0 Nov 01 06:52:45.574974 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:45.574811 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" Nov 01 06:52:45.574974 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:45.574848 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" event={"ID":"14b4da92-5164-45df-8306-5f228a7c5ac2","Type":"ContainerDied","Data":"61e42cad06057a6933769c825bb406fe1448b60b24169e0bcd8a699045509ed9"} Nov 01 06:52:45.574974 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:45.574903 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq" event={"ID":"14b4da92-5164-45df-8306-5f228a7c5ac2","Type":"ContainerDied","Data":"bc39b2531dfc52449ea4c5ea2c0dc6feb32d7ada5e42f678ed37167dc5ef5589"} Nov 01 06:52:45.574974 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:45.574926 5040 scope.go:122] "RemoveContainer" containerID="61e42cad06057a6933769c825bb406fe1448b60b24169e0bcd8a699045509ed9" Nov 01 06:52:45.584770 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:45.584748 5040 scope.go:122] "RemoveContainer" containerID="ff3ac7610ee3e1a638c58bed2ebc6b8eff09ecc403f1b740651aba81e65c6228" Nov 01 06:52:45.593716 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:45.593699 5040 scope.go:122] "RemoveContainer" containerID="61e42cad06057a6933769c825bb406fe1448b60b24169e0bcd8a699045509ed9" Nov 01 06:52:45.593970 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:52:45.593952 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61e42cad06057a6933769c825bb406fe1448b60b24169e0bcd8a699045509ed9\": container with ID starting with 61e42cad06057a6933769c825bb406fe1448b60b24169e0bcd8a699045509ed9 not found: ID does not exist" containerID="61e42cad06057a6933769c825bb406fe1448b60b24169e0bcd8a699045509ed9" Nov 01 06:52:45.594037 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:45.593978 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61e42cad06057a6933769c825bb406fe1448b60b24169e0bcd8a699045509ed9"} err="failed to get container status \"61e42cad06057a6933769c825bb406fe1448b60b24169e0bcd8a699045509ed9\": rpc error: code = NotFound desc = could not find container \"61e42cad06057a6933769c825bb406fe1448b60b24169e0bcd8a699045509ed9\": container with ID starting with 61e42cad06057a6933769c825bb406fe1448b60b24169e0bcd8a699045509ed9 not found: ID does not exist" Nov 01 06:52:45.594037 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:45.593994 5040 scope.go:122] "RemoveContainer" containerID="ff3ac7610ee3e1a638c58bed2ebc6b8eff09ecc403f1b740651aba81e65c6228" Nov 01 06:52:45.594228 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:52:45.594210 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff3ac7610ee3e1a638c58bed2ebc6b8eff09ecc403f1b740651aba81e65c6228\": container with ID starting with ff3ac7610ee3e1a638c58bed2ebc6b8eff09ecc403f1b740651aba81e65c6228 not found: ID does not exist" containerID="ff3ac7610ee3e1a638c58bed2ebc6b8eff09ecc403f1b740651aba81e65c6228" Nov 01 06:52:45.594271 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:45.594231 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff3ac7610ee3e1a638c58bed2ebc6b8eff09ecc403f1b740651aba81e65c6228"} err="failed to get container status \"ff3ac7610ee3e1a638c58bed2ebc6b8eff09ecc403f1b740651aba81e65c6228\": rpc error: code = NotFound desc = could not find container \"ff3ac7610ee3e1a638c58bed2ebc6b8eff09ecc403f1b740651aba81e65c6228\": container with ID starting with ff3ac7610ee3e1a638c58bed2ebc6b8eff09ecc403f1b740651aba81e65c6228 not found: ID does not exist" Nov 01 06:52:45.741812 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:45.741775 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq"] Nov 01 06:52:45.792753 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:45.792672 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-747b7b-openshift-default-d64cb854f-jvttq"] Nov 01 06:52:47.514108 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:47.514075 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14b4da92-5164-45df-8306-5f228a7c5ac2" path="/var/lib/kubelet/pods/14b4da92-5164-45df-8306-5f228a7c5ac2/volumes" Nov 01 06:52:47.735607 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:47.735570 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71cea4-589654b775-j76mx" podUID="785f90ff-95cf-4273-9952-6877d4ffb72d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:52:47.735771 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:47.735575 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71cea4-799667f47-zx5bs" podUID="232b8e8b-3616-4768-9741-50e93d149e4b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:52:54.048800 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:54.048759 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81edfced-fkjkq"] Nov 01 06:52:54.099306 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:54.099273 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81edfced-fkjkq"] Nov 01 06:52:55.515185 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:55.515148 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="486c7011-34d6-47b7-8936-b3fc0c6fd1a4" path="/var/lib/kubelet/pods/486c7011-34d6-47b7-8936-b3fc0c6fd1a4/volumes" Nov 01 06:52:57.735908 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:57.735864 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71cea4-799667f47-zx5bs" podUID="232b8e8b-3616-4768-9741-50e93d149e4b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:52:57.736409 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:52:57.735917 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71cea4-589654b775-j76mx" podUID="785f90ff-95cf-4273-9952-6877d4ffb72d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:53:00.116944 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:00.116914 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71cea4-799667f47-zx5bs" Nov 01 06:53:00.236435 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:00.235961 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/232b8e8b-3616-4768-9741-50e93d149e4b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/232b8e8b-3616-4768-9741-50e93d149e4b-plugins-config-volume\") pod \"232b8e8b-3616-4768-9741-50e93d149e4b\" (UID: \"232b8e8b-3616-4768-9741-50e93d149e4b\") " Nov 01 06:53:00.236435 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:00.236032 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/232b8e8b-3616-4768-9741-50e93d149e4b-kube-api-access-g9xrf\" (UniqueName: \"kubernetes.io/projected/232b8e8b-3616-4768-9741-50e93d149e4b-kube-api-access-g9xrf\") pod \"232b8e8b-3616-4768-9741-50e93d149e4b\" (UID: \"232b8e8b-3616-4768-9741-50e93d149e4b\") " Nov 01 06:53:00.236435 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:00.236410 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/232b8e8b-3616-4768-9741-50e93d149e4b-plugins-config-volume" pod "232b8e8b-3616-4768-9741-50e93d149e4b" (UID: "232b8e8b-3616-4768-9741-50e93d149e4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:53:00.238858 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:00.238823 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/232b8e8b-3616-4768-9741-50e93d149e4b-kube-api-access-g9xrf" pod "232b8e8b-3616-4768-9741-50e93d149e4b" (UID: "232b8e8b-3616-4768-9741-50e93d149e4b"). InnerVolumeSpecName "kube-api-access-g9xrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:53:00.337900 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:00.337863 5040 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/232b8e8b-3616-4768-9741-50e93d149e4b-plugins-config-volume\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:53:00.337900 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:00.337892 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9xrf\" (UniqueName: \"kubernetes.io/projected/232b8e8b-3616-4768-9741-50e93d149e4b-kube-api-access-g9xrf\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:53:00.672013 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:00.671980 5040 generic.go:356] "Generic (PLEG): container finished" podID="232b8e8b-3616-4768-9741-50e93d149e4b" containerID="98ca89d7a9974d487c924ddcc9c9299c19d85897a09a2e9a0d55a08a3acd69b2" exitCode=0 Nov 01 06:53:00.672215 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:00.672052 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71cea4-799667f47-zx5bs" Nov 01 06:53:00.672215 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:00.672069 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71cea4-799667f47-zx5bs" event={"ID":"232b8e8b-3616-4768-9741-50e93d149e4b","Type":"ContainerDied","Data":"98ca89d7a9974d487c924ddcc9c9299c19d85897a09a2e9a0d55a08a3acd69b2"} Nov 01 06:53:00.672215 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:00.672132 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71cea4-799667f47-zx5bs" event={"ID":"232b8e8b-3616-4768-9741-50e93d149e4b","Type":"ContainerDied","Data":"a491b05a6c31629566094151595b78bc921f063b4e1a9e36143f000f599f699d"} Nov 01 06:53:00.672215 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:00.672161 5040 scope.go:122] "RemoveContainer" containerID="98ca89d7a9974d487c924ddcc9c9299c19d85897a09a2e9a0d55a08a3acd69b2" Nov 01 06:53:00.674166 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:00.674147 5040 generic.go:356] "Generic (PLEG): container finished" podID="785f90ff-95cf-4273-9952-6877d4ffb72d" containerID="c35c43d2b8f46576a8609704f55e86afaae337c347947802b1e8b052934badc1" exitCode=0 Nov 01 06:53:00.674297 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:00.674219 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71cea4-589654b775-j76mx" event={"ID":"785f90ff-95cf-4273-9952-6877d4ffb72d","Type":"ContainerDied","Data":"c35c43d2b8f46576a8609704f55e86afaae337c347947802b1e8b052934badc1"} Nov 01 06:53:00.681725 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:00.681706 5040 scope.go:122] "RemoveContainer" containerID="98ca89d7a9974d487c924ddcc9c9299c19d85897a09a2e9a0d55a08a3acd69b2" Nov 01 06:53:00.682016 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:53:00.681996 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98ca89d7a9974d487c924ddcc9c9299c19d85897a09a2e9a0d55a08a3acd69b2\": container with ID starting with 98ca89d7a9974d487c924ddcc9c9299c19d85897a09a2e9a0d55a08a3acd69b2 not found: ID does not exist" containerID="98ca89d7a9974d487c924ddcc9c9299c19d85897a09a2e9a0d55a08a3acd69b2" Nov 01 06:53:00.682105 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:00.682020 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98ca89d7a9974d487c924ddcc9c9299c19d85897a09a2e9a0d55a08a3acd69b2"} err="failed to get container status \"98ca89d7a9974d487c924ddcc9c9299c19d85897a09a2e9a0d55a08a3acd69b2\": rpc error: code = NotFound desc = could not find container \"98ca89d7a9974d487c924ddcc9c9299c19d85897a09a2e9a0d55a08a3acd69b2\": container with ID starting with 98ca89d7a9974d487c924ddcc9c9299c19d85897a09a2e9a0d55a08a3acd69b2 not found: ID does not exist" Nov 01 06:53:00.759623 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:00.759587 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71cea4-589654b775-j76mx" Nov 01 06:53:00.839496 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:00.839403 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71cea4-799667f47-zx5bs"] Nov 01 06:53:00.843787 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:00.843760 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/785f90ff-95cf-4273-9952-6877d4ffb72d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/785f90ff-95cf-4273-9952-6877d4ffb72d-plugins-config-volume\") pod \"785f90ff-95cf-4273-9952-6877d4ffb72d\" (UID: \"785f90ff-95cf-4273-9952-6877d4ffb72d\") " Nov 01 06:53:00.843957 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:00.843848 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/785f90ff-95cf-4273-9952-6877d4ffb72d-kube-api-access-ttlpt\" (UniqueName: \"kubernetes.io/projected/785f90ff-95cf-4273-9952-6877d4ffb72d-kube-api-access-ttlpt\") pod \"785f90ff-95cf-4273-9952-6877d4ffb72d\" (UID: \"785f90ff-95cf-4273-9952-6877d4ffb72d\") " Nov 01 06:53:00.844194 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:00.844169 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/785f90ff-95cf-4273-9952-6877d4ffb72d-plugins-config-volume" pod "785f90ff-95cf-4273-9952-6877d4ffb72d" (UID: "785f90ff-95cf-4273-9952-6877d4ffb72d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:53:00.845914 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:00.845893 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/785f90ff-95cf-4273-9952-6877d4ffb72d-kube-api-access-ttlpt" pod "785f90ff-95cf-4273-9952-6877d4ffb72d" (UID: "785f90ff-95cf-4273-9952-6877d4ffb72d"). InnerVolumeSpecName "kube-api-access-ttlpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:53:00.890218 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:00.890187 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71cea4-799667f47-zx5bs"] Nov 01 06:53:00.944968 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:00.944935 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttlpt\" (UniqueName: \"kubernetes.io/projected/785f90ff-95cf-4273-9952-6877d4ffb72d-kube-api-access-ttlpt\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:53:00.944968 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:00.944966 5040 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/785f90ff-95cf-4273-9952-6877d4ffb72d-plugins-config-volume\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:53:01.513894 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:01.513859 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="232b8e8b-3616-4768-9741-50e93d149e4b" path="/var/lib/kubelet/pods/232b8e8b-3616-4768-9741-50e93d149e4b/volumes" Nov 01 06:53:01.681579 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:01.681549 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71cea4-589654b775-j76mx" Nov 01 06:53:01.681754 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:01.681584 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71cea4-589654b775-j76mx" event={"ID":"785f90ff-95cf-4273-9952-6877d4ffb72d","Type":"ContainerDied","Data":"a602025192242dd1868c7e857e068a65604315020179b541cae912b1bcaec39d"} Nov 01 06:53:01.681754 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:01.681644 5040 scope.go:122] "RemoveContainer" containerID="c35c43d2b8f46576a8609704f55e86afaae337c347947802b1e8b052934badc1" Nov 01 06:53:01.848208 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:01.848128 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-71cea4-589654b775-j76mx"] Nov 01 06:53:01.898794 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:01.898756 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-71cea4-589654b775-j76mx"] Nov 01 06:53:03.513921 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:03.513888 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="785f90ff-95cf-4273-9952-6877d4ffb72d" path="/var/lib/kubelet/pods/785f90ff-95cf-4273-9952-6877d4ffb72d/volumes" Nov 01 06:53:06.468028 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:06.467997 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf"] Nov 01 06:53:06.468463 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:06.468223 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" podUID="1c87f5c4-df00-4259-823d-b2d51418e42d" containerName="istio-proxy" containerID="cri-o://3eef6d7c23966931820cdbdef29c4123e74b60848d7585a09f8333b0e3f8724d" gracePeriod=30 Nov 01 06:53:09.972872 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:53:09.972831 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:53:10.249550 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:10.249512 5040 patch_prober.go:29] interesting pod/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 06:53:10.249707 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:10.249566 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" podUID="1c87f5c4-df00-4259-823d-b2d51418e42d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 06:53:11.721038 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.721007 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:53:11.747022 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.746982 5040 generic.go:356] "Generic (PLEG): container finished" podID="1c87f5c4-df00-4259-823d-b2d51418e42d" containerID="3eef6d7c23966931820cdbdef29c4123e74b60848d7585a09f8333b0e3f8724d" exitCode=0 Nov 01 06:53:11.747194 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.747037 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" Nov 01 06:53:11.747194 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.747070 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" event={"ID":"1c87f5c4-df00-4259-823d-b2d51418e42d","Type":"ContainerDied","Data":"3eef6d7c23966931820cdbdef29c4123e74b60848d7585a09f8333b0e3f8724d"} Nov 01 06:53:11.747194 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.747129 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf" event={"ID":"1c87f5c4-df00-4259-823d-b2d51418e42d","Type":"ContainerDied","Data":"16aad99dcd63246590c1a08248143e13b050390f3fe65cbec42ce1fa4fbcaa16"} Nov 01 06:53:11.747194 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.747156 5040 scope.go:122] "RemoveContainer" containerID="3eef6d7c23966931820cdbdef29c4123e74b60848d7585a09f8333b0e3f8724d" Nov 01 06:53:11.756717 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.756695 5040 scope.go:122] "RemoveContainer" containerID="df427a0991845a5191566cb4608352e8ed40c7af15df647bc01995cf48ba228f" Nov 01 06:53:11.765725 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.765700 5040 scope.go:122] "RemoveContainer" containerID="3eef6d7c23966931820cdbdef29c4123e74b60848d7585a09f8333b0e3f8724d" Nov 01 06:53:11.766067 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:53:11.766041 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eef6d7c23966931820cdbdef29c4123e74b60848d7585a09f8333b0e3f8724d\": container with ID starting with 3eef6d7c23966931820cdbdef29c4123e74b60848d7585a09f8333b0e3f8724d not found: ID does not exist" containerID="3eef6d7c23966931820cdbdef29c4123e74b60848d7585a09f8333b0e3f8724d" Nov 01 06:53:11.766178 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.766082 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eef6d7c23966931820cdbdef29c4123e74b60848d7585a09f8333b0e3f8724d"} err="failed to get container status \"3eef6d7c23966931820cdbdef29c4123e74b60848d7585a09f8333b0e3f8724d\": rpc error: code = NotFound desc = could not find container \"3eef6d7c23966931820cdbdef29c4123e74b60848d7585a09f8333b0e3f8724d\": container with ID starting with 3eef6d7c23966931820cdbdef29c4123e74b60848d7585a09f8333b0e3f8724d not found: ID does not exist" Nov 01 06:53:11.766178 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.766108 5040 scope.go:122] "RemoveContainer" containerID="df427a0991845a5191566cb4608352e8ed40c7af15df647bc01995cf48ba228f" Nov 01 06:53:11.766380 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:53:11.766359 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df427a0991845a5191566cb4608352e8ed40c7af15df647bc01995cf48ba228f\": container with ID starting with df427a0991845a5191566cb4608352e8ed40c7af15df647bc01995cf48ba228f not found: ID does not exist" containerID="df427a0991845a5191566cb4608352e8ed40c7af15df647bc01995cf48ba228f" Nov 01 06:53:11.766424 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.766388 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df427a0991845a5191566cb4608352e8ed40c7af15df647bc01995cf48ba228f"} err="failed to get container status \"df427a0991845a5191566cb4608352e8ed40c7af15df647bc01995cf48ba228f\": rpc error: code = NotFound desc = could not find container \"df427a0991845a5191566cb4608352e8ed40c7af15df647bc01995cf48ba228f\": container with ID starting with df427a0991845a5191566cb4608352e8ed40c7af15df647bc01995cf48ba228f not found: ID does not exist" Nov 01 06:53:11.843462 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.843425 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c87f5c4-df00-4259-823d-b2d51418e42d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c87f5c4-df00-4259-823d-b2d51418e42d-istio-data\") pod \"1c87f5c4-df00-4259-823d-b2d51418e42d\" (UID: \"1c87f5c4-df00-4259-823d-b2d51418e42d\") " Nov 01 06:53:11.843654 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.843467 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c87f5c4-df00-4259-823d-b2d51418e42d-istio-token\" (UniqueName: \"kubernetes.io/projected/1c87f5c4-df00-4259-823d-b2d51418e42d-istio-token\") pod \"1c87f5c4-df00-4259-823d-b2d51418e42d\" (UID: \"1c87f5c4-df00-4259-823d-b2d51418e42d\") " Nov 01 06:53:11.843654 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.843530 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c87f5c4-df00-4259-823d-b2d51418e42d-kube-api-access-fcxzr\" (UniqueName: \"kubernetes.io/projected/1c87f5c4-df00-4259-823d-b2d51418e42d-kube-api-access-fcxzr\") pod \"1c87f5c4-df00-4259-823d-b2d51418e42d\" (UID: \"1c87f5c4-df00-4259-823d-b2d51418e42d\") " Nov 01 06:53:11.843654 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.843571 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c87f5c4-df00-4259-823d-b2d51418e42d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c87f5c4-df00-4259-823d-b2d51418e42d-credential-socket\") pod \"1c87f5c4-df00-4259-823d-b2d51418e42d\" (UID: \"1c87f5c4-df00-4259-823d-b2d51418e42d\") " Nov 01 06:53:11.843654 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.843633 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c87f5c4-df00-4259-823d-b2d51418e42d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c87f5c4-df00-4259-823d-b2d51418e42d-workload-socket\") pod \"1c87f5c4-df00-4259-823d-b2d51418e42d\" (UID: \"1c87f5c4-df00-4259-823d-b2d51418e42d\") " Nov 01 06:53:11.843861 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.843675 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c87f5c4-df00-4259-823d-b2d51418e42d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c87f5c4-df00-4259-823d-b2d51418e42d-istio-podinfo\") pod \"1c87f5c4-df00-4259-823d-b2d51418e42d\" (UID: \"1c87f5c4-df00-4259-823d-b2d51418e42d\") " Nov 01 06:53:11.843861 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.843715 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c87f5c4-df00-4259-823d-b2d51418e42d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c87f5c4-df00-4259-823d-b2d51418e42d-workload-certs\") pod \"1c87f5c4-df00-4259-823d-b2d51418e42d\" (UID: \"1c87f5c4-df00-4259-823d-b2d51418e42d\") " Nov 01 06:53:11.843861 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.843745 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c87f5c4-df00-4259-823d-b2d51418e42d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c87f5c4-df00-4259-823d-b2d51418e42d-istiod-ca-cert\") pod \"1c87f5c4-df00-4259-823d-b2d51418e42d\" (UID: \"1c87f5c4-df00-4259-823d-b2d51418e42d\") " Nov 01 06:53:11.843861 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.843751 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c87f5c4-df00-4259-823d-b2d51418e42d-istio-data" pod "1c87f5c4-df00-4259-823d-b2d51418e42d" (UID: "1c87f5c4-df00-4259-823d-b2d51418e42d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:53:11.843861 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.843810 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c87f5c4-df00-4259-823d-b2d51418e42d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c87f5c4-df00-4259-823d-b2d51418e42d-istio-envoy\") pod \"1c87f5c4-df00-4259-823d-b2d51418e42d\" (UID: \"1c87f5c4-df00-4259-823d-b2d51418e42d\") " Nov 01 06:53:11.844150 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.843975 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c87f5c4-df00-4259-823d-b2d51418e42d-credential-socket" pod "1c87f5c4-df00-4259-823d-b2d51418e42d" (UID: "1c87f5c4-df00-4259-823d-b2d51418e42d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:53:11.844150 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.843984 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c87f5c4-df00-4259-823d-b2d51418e42d-workload-socket" pod "1c87f5c4-df00-4259-823d-b2d51418e42d" (UID: "1c87f5c4-df00-4259-823d-b2d51418e42d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:53:11.844150 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.844038 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c87f5c4-df00-4259-823d-b2d51418e42d-workload-certs" pod "1c87f5c4-df00-4259-823d-b2d51418e42d" (UID: "1c87f5c4-df00-4259-823d-b2d51418e42d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:53:11.844328 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.844172 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c87f5c4-df00-4259-823d-b2d51418e42d-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:53:11.844328 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.844191 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c87f5c4-df00-4259-823d-b2d51418e42d-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:53:11.844328 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.844206 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c87f5c4-df00-4259-823d-b2d51418e42d-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:53:11.844328 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.844220 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c87f5c4-df00-4259-823d-b2d51418e42d-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:53:11.844328 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.844275 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c87f5c4-df00-4259-823d-b2d51418e42d-istiod-ca-cert" pod "1c87f5c4-df00-4259-823d-b2d51418e42d" (UID: "1c87f5c4-df00-4259-823d-b2d51418e42d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:53:11.845855 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.845828 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c87f5c4-df00-4259-823d-b2d51418e42d-istio-token" pod "1c87f5c4-df00-4259-823d-b2d51418e42d" (UID: "1c87f5c4-df00-4259-823d-b2d51418e42d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:53:11.845991 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.845961 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c87f5c4-df00-4259-823d-b2d51418e42d-istio-envoy" pod "1c87f5c4-df00-4259-823d-b2d51418e42d" (UID: "1c87f5c4-df00-4259-823d-b2d51418e42d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:53:11.846048 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.845998 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c87f5c4-df00-4259-823d-b2d51418e42d-istio-podinfo" pod "1c87f5c4-df00-4259-823d-b2d51418e42d" (UID: "1c87f5c4-df00-4259-823d-b2d51418e42d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:53:11.846092 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.846069 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c87f5c4-df00-4259-823d-b2d51418e42d-kube-api-access-fcxzr" pod "1c87f5c4-df00-4259-823d-b2d51418e42d" (UID: "1c87f5c4-df00-4259-823d-b2d51418e42d"). InnerVolumeSpecName "kube-api-access-fcxzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:53:11.944883 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.944850 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c87f5c4-df00-4259-823d-b2d51418e42d-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:53:11.944883 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.944880 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c87f5c4-df00-4259-823d-b2d51418e42d-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:53:11.945104 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.944891 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcxzr\" (UniqueName: \"kubernetes.io/projected/1c87f5c4-df00-4259-823d-b2d51418e42d-kube-api-access-fcxzr\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:53:11.945104 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.944902 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c87f5c4-df00-4259-823d-b2d51418e42d-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:53:11.945104 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:11.944913 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c87f5c4-df00-4259-823d-b2d51418e42d-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:53:12.170154 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:12.170114 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf"] Nov 01 06:53:12.173955 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:12.173926 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-71cea4-openshift-default-7b554798bf-6mdgf"] Nov 01 06:53:13.513661 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:13.513625 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c87f5c4-df00-4259-823d-b2d51418e42d" path="/var/lib/kubelet/pods/1c87f5c4-df00-4259-823d-b2d51418e42d/volumes" Nov 01 06:53:16.051913 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:16.051875 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d67d7f00-xmj9q"] Nov 01 06:53:16.101984 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:16.101947 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d67d7f00-xmj9q"] Nov 01 06:53:17.514083 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:17.514042 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb479226-2d4d-470f-9202-2ca474ad9f5c" path="/var/lib/kubelet/pods/fb479226-2d4d-470f-9202-2ca474ad9f5c/volumes" Nov 01 06:53:27.047177 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:27.047137 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c03cdb9-zld5v"] Nov 01 06:53:27.097380 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:27.097346 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c03cdb9-zld5v"] Nov 01 06:53:27.514740 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:27.514703 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c623288c-7e7e-413f-9ee4-7b7bc14212d8" path="/var/lib/kubelet/pods/c623288c-7e7e-413f-9ee4-7b7bc14212d8/volumes" Nov 01 06:53:28.261570 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:28.261531 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2"] Nov 01 06:53:28.262109 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:28.261737 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" podUID="932df59a-5bc9-4c43-a4ad-55dc8b516b7b" containerName="istio-proxy" containerID="cri-o://b305940539dfa8d3c0e5caa0765ac5c29d11eb7ee1d788aeeea9388dcae454a4" gracePeriod=30 Nov 01 06:53:33.517733 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.517707 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:53:33.539361 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.539287 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-istiod-ca-cert\") pod \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\" (UID: \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\") " Nov 01 06:53:33.539361 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.539327 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-istio-envoy\") pod \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\" (UID: \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\") " Nov 01 06:53:33.539619 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.539363 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-workload-certs\") pod \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\" (UID: \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\") " Nov 01 06:53:33.539619 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.539394 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-istio-podinfo\") pod \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\" (UID: \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\") " Nov 01 06:53:33.539619 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.539450 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-istio-data\") pod \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\" (UID: \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\") " Nov 01 06:53:33.539619 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.539514 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-istio-token\" (UniqueName: \"kubernetes.io/projected/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-istio-token\") pod \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\" (UID: \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\") " Nov 01 06:53:33.539619 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.539549 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-workload-socket\") pod \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\" (UID: \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\") " Nov 01 06:53:33.539619 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.539577 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-kube-api-access-52bj2\" (UniqueName: \"kubernetes.io/projected/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-kube-api-access-52bj2\") pod \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\" (UID: \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\") " Nov 01 06:53:33.539942 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.539634 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-credential-socket\") pod \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\" (UID: \"932df59a-5bc9-4c43-a4ad-55dc8b516b7b\") " Nov 01 06:53:33.539942 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.539706 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-workload-certs" pod "932df59a-5bc9-4c43-a4ad-55dc8b516b7b" (UID: "932df59a-5bc9-4c43-a4ad-55dc8b516b7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:53:33.539942 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.539784 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-istiod-ca-cert" pod "932df59a-5bc9-4c43-a4ad-55dc8b516b7b" (UID: "932df59a-5bc9-4c43-a4ad-55dc8b516b7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:53:33.540080 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.539973 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-credential-socket" pod "932df59a-5bc9-4c43-a4ad-55dc8b516b7b" (UID: "932df59a-5bc9-4c43-a4ad-55dc8b516b7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:53:33.540080 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.539984 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-workload-socket" pod "932df59a-5bc9-4c43-a4ad-55dc8b516b7b" (UID: "932df59a-5bc9-4c43-a4ad-55dc8b516b7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:53:33.540080 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.540006 5040 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-istiod-ca-cert\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:53:33.540080 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.540029 5040 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-workload-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:53:33.540080 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.540049 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-istio-data" pod "932df59a-5bc9-4c43-a4ad-55dc8b516b7b" (UID: "932df59a-5bc9-4c43-a4ad-55dc8b516b7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:53:33.541579 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.541550 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-istio-podinfo" pod "932df59a-5bc9-4c43-a4ad-55dc8b516b7b" (UID: "932df59a-5bc9-4c43-a4ad-55dc8b516b7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:53:33.541724 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.541559 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-istio-envoy" pod "932df59a-5bc9-4c43-a4ad-55dc8b516b7b" (UID: "932df59a-5bc9-4c43-a4ad-55dc8b516b7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:53:33.542073 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.542053 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-kube-api-access-52bj2" pod "932df59a-5bc9-4c43-a4ad-55dc8b516b7b" (UID: "932df59a-5bc9-4c43-a4ad-55dc8b516b7b"). InnerVolumeSpecName "kube-api-access-52bj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:53:33.542139 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.542070 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-istio-token" pod "932df59a-5bc9-4c43-a4ad-55dc8b516b7b" (UID: "932df59a-5bc9-4c43-a4ad-55dc8b516b7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:53:33.640491 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.640439 5040 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-credential-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:53:33.640491 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.640470 5040 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-istio-envoy\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:53:33.640491 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.640501 5040 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-istio-podinfo\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:53:33.640740 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.640511 5040 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-istio-data\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:53:33.640740 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.640520 5040 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-istio-token\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:53:33.640740 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.640528 5040 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-workload-socket\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:53:33.640740 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.640538 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52bj2\" (UniqueName: \"kubernetes.io/projected/932df59a-5bc9-4c43-a4ad-55dc8b516b7b-kube-api-access-52bj2\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:53:33.885513 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.885407 5040 generic.go:356] "Generic (PLEG): container finished" podID="932df59a-5bc9-4c43-a4ad-55dc8b516b7b" containerID="b305940539dfa8d3c0e5caa0765ac5c29d11eb7ee1d788aeeea9388dcae454a4" exitCode=0 Nov 01 06:53:33.885513 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.885458 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" Nov 01 06:53:33.885513 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.885503 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" event={"ID":"932df59a-5bc9-4c43-a4ad-55dc8b516b7b","Type":"ContainerDied","Data":"b305940539dfa8d3c0e5caa0765ac5c29d11eb7ee1d788aeeea9388dcae454a4"} Nov 01 06:53:33.885782 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.885557 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2" event={"ID":"932df59a-5bc9-4c43-a4ad-55dc8b516b7b","Type":"ContainerDied","Data":"cb7c873a7ca13bf3d2603410690ed3c467630379c73b89e9ce0d9254b9552c02"} Nov 01 06:53:33.885782 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.885579 5040 scope.go:122] "RemoveContainer" containerID="b305940539dfa8d3c0e5caa0765ac5c29d11eb7ee1d788aeeea9388dcae454a4" Nov 01 06:53:33.896190 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.896169 5040 scope.go:122] "RemoveContainer" containerID="cc8c7cd1fb428c388d3a659f53f1391db97c46c9e8f0f3b7dd1c45f8b5dba17c" Nov 01 06:53:33.905834 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.905816 5040 scope.go:122] "RemoveContainer" containerID="b305940539dfa8d3c0e5caa0765ac5c29d11eb7ee1d788aeeea9388dcae454a4" Nov 01 06:53:33.906102 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:53:33.906081 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b305940539dfa8d3c0e5caa0765ac5c29d11eb7ee1d788aeeea9388dcae454a4\": container with ID starting with b305940539dfa8d3c0e5caa0765ac5c29d11eb7ee1d788aeeea9388dcae454a4 not found: ID does not exist" containerID="b305940539dfa8d3c0e5caa0765ac5c29d11eb7ee1d788aeeea9388dcae454a4" Nov 01 06:53:33.906141 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.906113 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b305940539dfa8d3c0e5caa0765ac5c29d11eb7ee1d788aeeea9388dcae454a4"} err="failed to get container status \"b305940539dfa8d3c0e5caa0765ac5c29d11eb7ee1d788aeeea9388dcae454a4\": rpc error: code = NotFound desc = could not find container \"b305940539dfa8d3c0e5caa0765ac5c29d11eb7ee1d788aeeea9388dcae454a4\": container with ID starting with b305940539dfa8d3c0e5caa0765ac5c29d11eb7ee1d788aeeea9388dcae454a4 not found: ID does not exist" Nov 01 06:53:33.906141 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.906132 5040 scope.go:122] "RemoveContainer" containerID="cc8c7cd1fb428c388d3a659f53f1391db97c46c9e8f0f3b7dd1c45f8b5dba17c" Nov 01 06:53:33.906384 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:53:33.906360 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc8c7cd1fb428c388d3a659f53f1391db97c46c9e8f0f3b7dd1c45f8b5dba17c\": container with ID starting with cc8c7cd1fb428c388d3a659f53f1391db97c46c9e8f0f3b7dd1c45f8b5dba17c not found: ID does not exist" containerID="cc8c7cd1fb428c388d3a659f53f1391db97c46c9e8f0f3b7dd1c45f8b5dba17c" Nov 01 06:53:33.906502 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:33.906389 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc8c7cd1fb428c388d3a659f53f1391db97c46c9e8f0f3b7dd1c45f8b5dba17c"} err="failed to get container status \"cc8c7cd1fb428c388d3a659f53f1391db97c46c9e8f0f3b7dd1c45f8b5dba17c\": rpc error: code = NotFound desc = could not find container \"cc8c7cd1fb428c388d3a659f53f1391db97c46c9e8f0f3b7dd1c45f8b5dba17c\": container with ID starting with cc8c7cd1fb428c388d3a659f53f1391db97c46c9e8f0f3b7dd1c45f8b5dba17c not found: ID does not exist" Nov 01 06:53:34.053112 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:34.053071 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2"] Nov 01 06:53:34.057508 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:34.057456 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-cfe368-openshift-default-fbb746fd5-lffg2"] Nov 01 06:53:35.513893 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:35.513861 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="932df59a-5bc9-4c43-a4ad-55dc8b516b7b" path="/var/lib/kubelet/pods/932df59a-5bc9-4c43-a4ad-55dc8b516b7b/volumes" Nov 01 06:53:39.972766 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:53:39.972721 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:53:41.429160 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:41.429124 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc"] Nov 01 06:53:41.480977 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:41.480940 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq"] Nov 01 06:53:56.448983 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.448938 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc" podUID="66fdf9da-4790-40c5-9434-052c7465d23e" containerName="main" containerID="cri-o://af58ca6fbb09563c9b43340b1a852a6aa9259ce988e71d84054eee5b812dcfad" gracePeriod=45 Nov 01 06:53:56.475537 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.475500 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc" podUID="66fdf9da-4790-40c5-9434-052c7465d23e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.124:8000/ready\": dial tcp 10.132.0.124:8000: connect: connection refused" Nov 01 06:53:56.500213 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.500164 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" podUID="5daa03ca-824b-4799-a11b-4b3af1512c38" containerName="main" containerID="cri-o://2b56028a64eec80ff09aa3ce9ffca497844c5ff3cf39634cbe2e6c6e7026823e" gracePeriod=45 Nov 01 06:53:56.752338 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.752301 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc" Nov 01 06:53:56.795084 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.795058 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" Nov 01 06:53:56.837444 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.837408 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5daa03ca-824b-4799-a11b-4b3af1512c38-tls-certs\" (UniqueName: \"kubernetes.io/secret/5daa03ca-824b-4799-a11b-4b3af1512c38-tls-certs\") pod \"5daa03ca-824b-4799-a11b-4b3af1512c38\" (UID: \"5daa03ca-824b-4799-a11b-4b3af1512c38\") " Nov 01 06:53:56.837666 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.837459 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66fdf9da-4790-40c5-9434-052c7465d23e-kube-api-access-dbcft\" (UniqueName: \"kubernetes.io/projected/66fdf9da-4790-40c5-9434-052c7465d23e-kube-api-access-dbcft\") pod \"66fdf9da-4790-40c5-9434-052c7465d23e\" (UID: \"66fdf9da-4790-40c5-9434-052c7465d23e\") " Nov 01 06:53:56.837666 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.837514 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5daa03ca-824b-4799-a11b-4b3af1512c38-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5daa03ca-824b-4799-a11b-4b3af1512c38-tmp-dir\") pod \"5daa03ca-824b-4799-a11b-4b3af1512c38\" (UID: \"5daa03ca-824b-4799-a11b-4b3af1512c38\") " Nov 01 06:53:56.837666 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.837540 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66fdf9da-4790-40c5-9434-052c7465d23e-tls-certs\" (UniqueName: \"kubernetes.io/secret/66fdf9da-4790-40c5-9434-052c7465d23e-tls-certs\") pod \"66fdf9da-4790-40c5-9434-052c7465d23e\" (UID: \"66fdf9da-4790-40c5-9434-052c7465d23e\") " Nov 01 06:53:56.837666 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.837573 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66fdf9da-4790-40c5-9434-052c7465d23e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/66fdf9da-4790-40c5-9434-052c7465d23e-dshm\") pod \"66fdf9da-4790-40c5-9434-052c7465d23e\" (UID: \"66fdf9da-4790-40c5-9434-052c7465d23e\") " Nov 01 06:53:56.837666 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.837609 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66fdf9da-4790-40c5-9434-052c7465d23e-home\" (UniqueName: \"kubernetes.io/empty-dir/66fdf9da-4790-40c5-9434-052c7465d23e-home\") pod \"66fdf9da-4790-40c5-9434-052c7465d23e\" (UID: \"66fdf9da-4790-40c5-9434-052c7465d23e\") " Nov 01 06:53:56.837666 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.837640 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5daa03ca-824b-4799-a11b-4b3af1512c38-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5daa03ca-824b-4799-a11b-4b3af1512c38-model-cache\") pod \"5daa03ca-824b-4799-a11b-4b3af1512c38\" (UID: \"5daa03ca-824b-4799-a11b-4b3af1512c38\") " Nov 01 06:53:56.838022 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.837687 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5daa03ca-824b-4799-a11b-4b3af1512c38-kube-api-access-dvzj9\" (UniqueName: \"kubernetes.io/projected/5daa03ca-824b-4799-a11b-4b3af1512c38-kube-api-access-dvzj9\") pod \"5daa03ca-824b-4799-a11b-4b3af1512c38\" (UID: \"5daa03ca-824b-4799-a11b-4b3af1512c38\") " Nov 01 06:53:56.838022 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.837743 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5daa03ca-824b-4799-a11b-4b3af1512c38-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5daa03ca-824b-4799-a11b-4b3af1512c38-dshm\") pod \"5daa03ca-824b-4799-a11b-4b3af1512c38\" (UID: \"5daa03ca-824b-4799-a11b-4b3af1512c38\") " Nov 01 06:53:56.838022 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.837814 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5daa03ca-824b-4799-a11b-4b3af1512c38-home\" (UniqueName: \"kubernetes.io/empty-dir/5daa03ca-824b-4799-a11b-4b3af1512c38-home\") pod \"5daa03ca-824b-4799-a11b-4b3af1512c38\" (UID: \"5daa03ca-824b-4799-a11b-4b3af1512c38\") " Nov 01 06:53:56.838022 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.837860 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66fdf9da-4790-40c5-9434-052c7465d23e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66fdf9da-4790-40c5-9434-052c7465d23e-model-cache\") pod \"66fdf9da-4790-40c5-9434-052c7465d23e\" (UID: \"66fdf9da-4790-40c5-9434-052c7465d23e\") " Nov 01 06:53:56.838022 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.837879 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5daa03ca-824b-4799-a11b-4b3af1512c38-tmp-dir" pod "5daa03ca-824b-4799-a11b-4b3af1512c38" (UID: "5daa03ca-824b-4799-a11b-4b3af1512c38"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:53:56.838022 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.837894 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66fdf9da-4790-40c5-9434-052c7465d23e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66fdf9da-4790-40c5-9434-052c7465d23e-tmp-dir\") pod \"66fdf9da-4790-40c5-9434-052c7465d23e\" (UID: \"66fdf9da-4790-40c5-9434-052c7465d23e\") " Nov 01 06:53:56.838022 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.837915 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66fdf9da-4790-40c5-9434-052c7465d23e-home" pod "66fdf9da-4790-40c5-9434-052c7465d23e" (UID: "66fdf9da-4790-40c5-9434-052c7465d23e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:53:56.838022 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.837958 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5daa03ca-824b-4799-a11b-4b3af1512c38-model-cache" pod "5daa03ca-824b-4799-a11b-4b3af1512c38" (UID: "5daa03ca-824b-4799-a11b-4b3af1512c38"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:53:56.838426 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.838093 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66fdf9da-4790-40c5-9434-052c7465d23e-tmp-dir" pod "66fdf9da-4790-40c5-9434-052c7465d23e" (UID: "66fdf9da-4790-40c5-9434-052c7465d23e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:53:56.838426 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.838234 5040 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66fdf9da-4790-40c5-9434-052c7465d23e-home\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:53:56.838426 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.838255 5040 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5daa03ca-824b-4799-a11b-4b3af1512c38-model-cache\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:53:56.838426 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.838268 5040 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66fdf9da-4790-40c5-9434-052c7465d23e-tmp-dir\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:53:56.838426 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.838271 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5daa03ca-824b-4799-a11b-4b3af1512c38-home" pod "5daa03ca-824b-4799-a11b-4b3af1512c38" (UID: "5daa03ca-824b-4799-a11b-4b3af1512c38"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:53:56.838426 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.838282 5040 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5daa03ca-824b-4799-a11b-4b3af1512c38-tmp-dir\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:53:56.838787 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.838455 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66fdf9da-4790-40c5-9434-052c7465d23e-model-cache" pod "66fdf9da-4790-40c5-9434-052c7465d23e" (UID: "66fdf9da-4790-40c5-9434-052c7465d23e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:53:56.840105 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.840071 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5daa03ca-824b-4799-a11b-4b3af1512c38-tls-certs" pod "5daa03ca-824b-4799-a11b-4b3af1512c38" (UID: "5daa03ca-824b-4799-a11b-4b3af1512c38"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:53:56.840239 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.840184 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5daa03ca-824b-4799-a11b-4b3af1512c38-kube-api-access-dvzj9" pod "5daa03ca-824b-4799-a11b-4b3af1512c38" (UID: "5daa03ca-824b-4799-a11b-4b3af1512c38"). InnerVolumeSpecName "kube-api-access-dvzj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:53:56.840239 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.840221 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66fdf9da-4790-40c5-9434-052c7465d23e-dshm" pod "66fdf9da-4790-40c5-9434-052c7465d23e" (UID: "66fdf9da-4790-40c5-9434-052c7465d23e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:53:56.840354 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.840228 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5daa03ca-824b-4799-a11b-4b3af1512c38-dshm" pod "5daa03ca-824b-4799-a11b-4b3af1512c38" (UID: "5daa03ca-824b-4799-a11b-4b3af1512c38"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:53:56.840354 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.840278 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66fdf9da-4790-40c5-9434-052c7465d23e-tls-certs" pod "66fdf9da-4790-40c5-9434-052c7465d23e" (UID: "66fdf9da-4790-40c5-9434-052c7465d23e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:53:56.840789 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.840771 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66fdf9da-4790-40c5-9434-052c7465d23e-kube-api-access-dbcft" pod "66fdf9da-4790-40c5-9434-052c7465d23e" (UID: "66fdf9da-4790-40c5-9434-052c7465d23e"). InnerVolumeSpecName "kube-api-access-dbcft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:53:56.939095 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.939062 5040 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5daa03ca-824b-4799-a11b-4b3af1512c38-dshm\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:53:56.939095 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.939092 5040 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5daa03ca-824b-4799-a11b-4b3af1512c38-home\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:53:56.939095 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.939102 5040 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66fdf9da-4790-40c5-9434-052c7465d23e-model-cache\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:53:56.939369 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.939112 5040 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5daa03ca-824b-4799-a11b-4b3af1512c38-tls-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:53:56.939369 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.939121 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbcft\" (UniqueName: \"kubernetes.io/projected/66fdf9da-4790-40c5-9434-052c7465d23e-kube-api-access-dbcft\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:53:56.939369 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.939133 5040 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66fdf9da-4790-40c5-9434-052c7465d23e-tls-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:53:56.939369 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.939141 5040 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66fdf9da-4790-40c5-9434-052c7465d23e-dshm\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:53:56.939369 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:56.939150 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvzj9\" (UniqueName: \"kubernetes.io/projected/5daa03ca-824b-4799-a11b-4b3af1512c38-kube-api-access-dvzj9\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:53:57.021601 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:57.021556 5040 generic.go:356] "Generic (PLEG): container finished" podID="66fdf9da-4790-40c5-9434-052c7465d23e" containerID="af58ca6fbb09563c9b43340b1a852a6aa9259ce988e71d84054eee5b812dcfad" exitCode=0 Nov 01 06:53:57.021819 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:57.021606 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc" Nov 01 06:53:57.021819 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:57.021617 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc" event={"ID":"66fdf9da-4790-40c5-9434-052c7465d23e","Type":"ContainerDied","Data":"af58ca6fbb09563c9b43340b1a852a6aa9259ce988e71d84054eee5b812dcfad"} Nov 01 06:53:57.021819 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:57.021684 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc" event={"ID":"66fdf9da-4790-40c5-9434-052c7465d23e","Type":"ContainerDied","Data":"912450bc3f6692e3e119f7795d739290f2007e2c3aeebf6e5fb910e5c025c3d9"} Nov 01 06:53:57.021819 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:57.021713 5040 scope.go:122] "RemoveContainer" containerID="af58ca6fbb09563c9b43340b1a852a6aa9259ce988e71d84054eee5b812dcfad" Nov 01 06:53:57.023750 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:57.023658 5040 generic.go:356] "Generic (PLEG): container finished" podID="5daa03ca-824b-4799-a11b-4b3af1512c38" containerID="2b56028a64eec80ff09aa3ce9ffca497844c5ff3cf39634cbe2e6c6e7026823e" exitCode=0 Nov 01 06:53:57.023750 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:57.023688 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" event={"ID":"5daa03ca-824b-4799-a11b-4b3af1512c38","Type":"ContainerDied","Data":"2b56028a64eec80ff09aa3ce9ffca497844c5ff3cf39634cbe2e6c6e7026823e"} Nov 01 06:53:57.023750 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:57.023747 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" event={"ID":"5daa03ca-824b-4799-a11b-4b3af1512c38","Type":"ContainerDied","Data":"28f135eef831d1b96eb4b72a9e5fb689928c8c8b87a2a839c9026387d6178aae"} Nov 01 06:53:57.023995 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:57.023695 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq" Nov 01 06:53:57.034991 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:57.034965 5040 scope.go:122] "RemoveContainer" containerID="af58ca6fbb09563c9b43340b1a852a6aa9259ce988e71d84054eee5b812dcfad" Nov 01 06:53:57.035413 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:53:57.035387 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af58ca6fbb09563c9b43340b1a852a6aa9259ce988e71d84054eee5b812dcfad\": container with ID starting with af58ca6fbb09563c9b43340b1a852a6aa9259ce988e71d84054eee5b812dcfad not found: ID does not exist" containerID="af58ca6fbb09563c9b43340b1a852a6aa9259ce988e71d84054eee5b812dcfad" Nov 01 06:53:57.035535 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:57.035426 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af58ca6fbb09563c9b43340b1a852a6aa9259ce988e71d84054eee5b812dcfad"} err="failed to get container status \"af58ca6fbb09563c9b43340b1a852a6aa9259ce988e71d84054eee5b812dcfad\": rpc error: code = NotFound desc = could not find container \"af58ca6fbb09563c9b43340b1a852a6aa9259ce988e71d84054eee5b812dcfad\": container with ID starting with af58ca6fbb09563c9b43340b1a852a6aa9259ce988e71d84054eee5b812dcfad not found: ID does not exist" Nov 01 06:53:57.035535 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:57.035451 5040 scope.go:122] "RemoveContainer" containerID="2b56028a64eec80ff09aa3ce9ffca497844c5ff3cf39634cbe2e6c6e7026823e" Nov 01 06:53:57.044752 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:57.044730 5040 scope.go:122] "RemoveContainer" containerID="2b56028a64eec80ff09aa3ce9ffca497844c5ff3cf39634cbe2e6c6e7026823e" Nov 01 06:53:57.045092 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:53:57.045066 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b56028a64eec80ff09aa3ce9ffca497844c5ff3cf39634cbe2e6c6e7026823e\": container with ID starting with 2b56028a64eec80ff09aa3ce9ffca497844c5ff3cf39634cbe2e6c6e7026823e not found: ID does not exist" containerID="2b56028a64eec80ff09aa3ce9ffca497844c5ff3cf39634cbe2e6c6e7026823e" Nov 01 06:53:57.045153 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:57.045107 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b56028a64eec80ff09aa3ce9ffca497844c5ff3cf39634cbe2e6c6e7026823e"} err="failed to get container status \"2b56028a64eec80ff09aa3ce9ffca497844c5ff3cf39634cbe2e6c6e7026823e\": rpc error: code = NotFound desc = could not find container \"2b56028a64eec80ff09aa3ce9ffca497844c5ff3cf39634cbe2e6c6e7026823e\": container with ID starting with 2b56028a64eec80ff09aa3ce9ffca497844c5ff3cf39634cbe2e6c6e7026823e not found: ID does not exist" Nov 01 06:53:57.188799 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:57.188763 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq"] Nov 01 06:53:57.195021 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:57.194993 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-9c8fc5/distinct-w0-9c8fc5-kserve-5cfc8cc64c-w8skq"] Nov 01 06:53:57.314576 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:57.314470 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc"] Nov 01 06:53:57.318859 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:57.318827 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-9c8fc5/distinct-2-w0-9c8fc5-kserve-644df67868-d2ghc"] Nov 01 06:53:57.515629 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:57.515588 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5daa03ca-824b-4799-a11b-4b3af1512c38" path="/var/lib/kubelet/pods/5daa03ca-824b-4799-a11b-4b3af1512c38/volumes" Nov 01 06:53:57.516039 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:53:57.516011 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66fdf9da-4790-40c5-9434-052c7465d23e" path="/var/lib/kubelet/pods/66fdf9da-4790-40c5-9434-052c7465d23e/volumes" Nov 01 06:54:09.971919 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:54:09.971883 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:54:20.050026 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:54:20.049989 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dc4bced-jxvvg"] Nov 01 06:54:20.100197 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:54:20.100156 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dc4bced-jxvvg"] Nov 01 06:54:21.515896 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:54:21.515796 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aef17fca-e57d-40af-bc03-88322b9d508e" path="/var/lib/kubelet/pods/aef17fca-e57d-40af-bc03-88322b9d508e/volumes" Nov 01 06:54:24.155635 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:54:24.155601 5040 scope.go:122] "RemoveContainer" containerID="88436aa3a5125f8c79098479d5b4f0f325c459d4d7de7711f8a361e87b259698" Nov 01 06:54:24.164721 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:54:24.164700 5040 scope.go:122] "RemoveContainer" containerID="ad7e9a907e506659e1dd815583c5764e28958ecca56699633903b1b20a2ccac9" Nov 01 06:54:24.173223 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:54:24.173203 5040 scope.go:122] "RemoveContainer" containerID="96fbfaa951350ee153e7e11e1dad29df612f3ae5e1e08a1f71b0636a627fccbf" Nov 01 06:54:24.183060 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:54:24.183040 5040 scope.go:122] "RemoveContainer" containerID="20fa5ab762cb05b47614d8bbc6fc748cb352f13024679cf7570147924df1f700" Nov 01 06:54:31.057606 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:54:31.057570 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01a4e934-pwrh4"] Nov 01 06:54:31.107794 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:54:31.107759 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01a4e934-pwrh4"] Nov 01 06:54:31.514338 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:54:31.514305 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac0d876d-4d31-4b49-b5ed-98dce05ee7a0" path="/var/lib/kubelet/pods/ac0d876d-4d31-4b49-b5ed-98dce05ee7a0/volumes" Nov 01 06:54:31.747413 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:54:31.747374 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-558f494858-fkgrf_ef1482fa-e45b-4784-939f-f6e9aa480c0e/authorino/0.log" Nov 01 06:54:32.128078 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:54:32.128033 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-p4xhq_4e64dd96-b5ba-4115-b87d-ffa91d637530/manager/0.log" Nov 01 06:54:32.378224 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:54:32.378142 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-6c4qz_da0c8971-758e-4a48-9f10-4c90ed9b4f1a/manager/0.log" Nov 01 06:54:39.972143 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:54:39.972101 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:54:44.607145 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:54:44.607103 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-p8pjd_58847028-6023-4f04-bd0d-9d6a4327b694/manager/0.log" Nov 01 06:54:46.840827 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:54:46.840794 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-788tv_72a66fbc-a5fe-42f7-8362-fe8e2a62025f/manager/1.log" Nov 01 06:54:48.150356 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:54:48.150317 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-c24v5_d73d9f29-4a48-46bf-8352-868e7d913a1b/manager/0.log" Nov 01 06:54:48.418128 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:54:48.418034 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-x2lbs_1edd6e44-240f-4f06-9294-f04fe4fcda65/proxy/0.log" Nov 01 06:54:49.148578 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:54:49.148537 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-788tv_72a66fbc-a5fe-42f7-8362-fe8e2a62025f/manager/0.log" Nov 01 06:55:09.971553 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:55:09.971517 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:55:22.046394 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:22.046358 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e43d1ae3-nglck"] Nov 01 06:55:22.096489 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:22.096441 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e43d1ae3-nglck"] Nov 01 06:55:22.289945 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:22.289912 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 06:55:22.301539 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:22.301444 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 06:55:23.513771 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:23.513738 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c3da87f-7ffc-4057-a412-80a547f8099b" path="/var/lib/kubelet/pods/8c3da87f-7ffc-4057-a412-80a547f8099b/volumes" Nov 01 06:55:24.229972 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:24.229938 5040 scope.go:122] "RemoveContainer" containerID="2a7ab790ba66b2b8128aac6790e41ba2d2f668797886fda5e77137aa912fcfb0" Nov 01 06:55:27.625677 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.625639 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4"] Nov 01 06:55:27.626108 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626054 5040 cpu_manager.go:414] "containerMap: removing container" podUID="14b4da92-5164-45df-8306-5f228a7c5ac2" containerName="istio-proxy" Nov 01 06:55:27.626108 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626071 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14b4da92-5164-45df-8306-5f228a7c5ac2" containerName="istio-proxy" Nov 01 06:55:27.626108 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626085 5040 cpu_manager.go:414] "containerMap: removing container" podUID="bb482d61-0774-4394-8c37-a564ef5b6838" containerName="istio-proxy" Nov 01 06:55:27.626108 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626093 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb482d61-0774-4394-8c37-a564ef5b6838" containerName="istio-proxy" Nov 01 06:55:27.626108 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626104 5040 cpu_manager.go:414] "containerMap: removing container" podUID="1fbb1642-46b7-4489-bdac-2075d11d644e" containerName="main" Nov 01 06:55:27.626108 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626110 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fbb1642-46b7-4489-bdac-2075d11d644e" containerName="main" Nov 01 06:55:27.626348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626118 5040 cpu_manager.go:414] "containerMap: removing container" podUID="5daa03ca-824b-4799-a11b-4b3af1512c38" containerName="main" Nov 01 06:55:27.626348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626123 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5daa03ca-824b-4799-a11b-4b3af1512c38" containerName="main" Nov 01 06:55:27.626348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626131 5040 cpu_manager.go:414] "containerMap: removing container" podUID="932df59a-5bc9-4c43-a4ad-55dc8b516b7b" containerName="istio-proxy" Nov 01 06:55:27.626348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626136 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="932df59a-5bc9-4c43-a4ad-55dc8b516b7b" containerName="istio-proxy" Nov 01 06:55:27.626348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626144 5040 cpu_manager.go:414] "containerMap: removing container" podUID="1c87f5c4-df00-4259-823d-b2d51418e42d" containerName="istio-proxy" Nov 01 06:55:27.626348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626149 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c87f5c4-df00-4259-823d-b2d51418e42d" containerName="istio-proxy" Nov 01 06:55:27.626348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626155 5040 cpu_manager.go:414] "containerMap: removing container" podUID="1c87f5c4-df00-4259-823d-b2d51418e42d" containerName="istio-proxy" Nov 01 06:55:27.626348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626160 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c87f5c4-df00-4259-823d-b2d51418e42d" containerName="istio-proxy" Nov 01 06:55:27.626348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626169 5040 cpu_manager.go:414] "containerMap: removing container" podUID="bb482d61-0774-4394-8c37-a564ef5b6838" containerName="istio-proxy" Nov 01 06:55:27.626348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626174 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb482d61-0774-4394-8c37-a564ef5b6838" containerName="istio-proxy" Nov 01 06:55:27.626348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626181 5040 cpu_manager.go:414] "containerMap: removing container" podUID="232b8e8b-3616-4768-9741-50e93d149e4b" containerName="payload-pre-processing" Nov 01 06:55:27.626348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626205 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="232b8e8b-3616-4768-9741-50e93d149e4b" containerName="payload-pre-processing" Nov 01 06:55:27.626348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626213 5040 cpu_manager.go:414] "containerMap: removing container" podUID="cee73e53-f9e2-49c9-a19c-ee33cc2ac68e" containerName="istio-proxy" Nov 01 06:55:27.626348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626220 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cee73e53-f9e2-49c9-a19c-ee33cc2ac68e" containerName="istio-proxy" Nov 01 06:55:27.626348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626231 5040 cpu_manager.go:414] "containerMap: removing container" podUID="cee73e53-f9e2-49c9-a19c-ee33cc2ac68e" containerName="istio-proxy" Nov 01 06:55:27.626348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626238 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cee73e53-f9e2-49c9-a19c-ee33cc2ac68e" containerName="istio-proxy" Nov 01 06:55:27.626348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626247 5040 cpu_manager.go:414] "containerMap: removing container" podUID="bfa4234e-3def-4d10-b181-c675e28e1b36" containerName="main" Nov 01 06:55:27.626348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626254 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfa4234e-3def-4d10-b181-c675e28e1b36" containerName="main" Nov 01 06:55:27.626348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626267 5040 cpu_manager.go:414] "containerMap: removing container" podUID="3381c8ac-63b7-4df0-b3f5-88579e89489e" containerName="revoke-keys" Nov 01 06:55:27.626348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626275 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3381c8ac-63b7-4df0-b3f5-88579e89489e" containerName="revoke-keys" Nov 01 06:55:27.626348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626283 5040 cpu_manager.go:414] "containerMap: removing container" podUID="66fdf9da-4790-40c5-9434-052c7465d23e" containerName="main" Nov 01 06:55:27.626348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626290 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66fdf9da-4790-40c5-9434-052c7465d23e" containerName="main" Nov 01 06:55:27.626348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626309 5040 cpu_manager.go:414] "containerMap: removing container" podUID="785f90ff-95cf-4273-9952-6877d4ffb72d" containerName="payload-processing" Nov 01 06:55:27.626348 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626316 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="785f90ff-95cf-4273-9952-6877d4ffb72d" containerName="payload-processing" Nov 01 06:55:27.627161 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626411 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="932df59a-5bc9-4c43-a4ad-55dc8b516b7b" containerName="istio-proxy" Nov 01 06:55:27.627161 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626424 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c87f5c4-df00-4259-823d-b2d51418e42d" containerName="istio-proxy" Nov 01 06:55:27.627161 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626435 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c87f5c4-df00-4259-823d-b2d51418e42d" containerName="istio-proxy" Nov 01 06:55:27.627161 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626447 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="14b4da92-5164-45df-8306-5f228a7c5ac2" containerName="istio-proxy" Nov 01 06:55:27.627161 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626454 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb482d61-0774-4394-8c37-a564ef5b6838" containerName="istio-proxy" Nov 01 06:55:27.627161 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626460 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="232b8e8b-3616-4768-9741-50e93d149e4b" containerName="payload-pre-processing" Nov 01 06:55:27.627161 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626465 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="cee73e53-f9e2-49c9-a19c-ee33cc2ac68e" containerName="istio-proxy" Nov 01 06:55:27.627161 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626470 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="932df59a-5bc9-4c43-a4ad-55dc8b516b7b" containerName="istio-proxy" Nov 01 06:55:27.627161 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626492 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="cee73e53-f9e2-49c9-a19c-ee33cc2ac68e" containerName="istio-proxy" Nov 01 06:55:27.627161 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626501 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfa4234e-3def-4d10-b181-c675e28e1b36" containerName="main" Nov 01 06:55:27.627161 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626510 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="3381c8ac-63b7-4df0-b3f5-88579e89489e" containerName="revoke-keys" Nov 01 06:55:27.627161 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626516 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="66fdf9da-4790-40c5-9434-052c7465d23e" containerName="main" Nov 01 06:55:27.627161 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626523 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="785f90ff-95cf-4273-9952-6877d4ffb72d" containerName="payload-processing" Nov 01 06:55:27.627161 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626531 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb482d61-0774-4394-8c37-a564ef5b6838" containerName="istio-proxy" Nov 01 06:55:27.627161 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626536 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fbb1642-46b7-4489-bdac-2075d11d644e" containerName="main" Nov 01 06:55:27.627161 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.626541 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="5daa03ca-824b-4799-a11b-4b3af1512c38" containerName="main" Nov 01 06:55:27.630905 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.630884 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4" Nov 01 06:55:27.634451 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.634425 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4hln\" (UniqueName: \"kubernetes.io/projected/d1211d6c-1cd3-4b33-a2a1-28125b936f91-kube-api-access-w4hln\") pod \"simulator-main-045f69-kserve-75c7b5f8cf-wn9h4\" (UID: \"d1211d6c-1cd3-4b33-a2a1-28125b936f91\") " pod="e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4" Nov 01 06:55:27.634579 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.634501 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1211d6c-1cd3-4b33-a2a1-28125b936f91-tls-certs\") pod \"simulator-main-045f69-kserve-75c7b5f8cf-wn9h4\" (UID: \"d1211d6c-1cd3-4b33-a2a1-28125b936f91\") " pod="e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4" Nov 01 06:55:27.634632 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.634588 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1211d6c-1cd3-4b33-a2a1-28125b936f91-home\") pod \"simulator-main-045f69-kserve-75c7b5f8cf-wn9h4\" (UID: \"d1211d6c-1cd3-4b33-a2a1-28125b936f91\") " pod="e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4" Nov 01 06:55:27.634632 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.634615 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1211d6c-1cd3-4b33-a2a1-28125b936f91-model-cache\") pod \"simulator-main-045f69-kserve-75c7b5f8cf-wn9h4\" (UID: \"d1211d6c-1cd3-4b33-a2a1-28125b936f91\") " pod="e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4" Nov 01 06:55:27.634709 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.634650 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1211d6c-1cd3-4b33-a2a1-28125b936f91-tmp-dir\") pod \"simulator-main-045f69-kserve-75c7b5f8cf-wn9h4\" (UID: \"d1211d6c-1cd3-4b33-a2a1-28125b936f91\") " pod="e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4" Nov 01 06:55:27.634758 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.634736 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1211d6c-1cd3-4b33-a2a1-28125b936f91-dshm\") pod \"simulator-main-045f69-kserve-75c7b5f8cf-wn9h4\" (UID: \"d1211d6c-1cd3-4b33-a2a1-28125b936f91\") " pod="e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4" Nov 01 06:55:27.716589 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.716553 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4"] Nov 01 06:55:27.735417 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.735380 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1211d6c-1cd3-4b33-a2a1-28125b936f91-tmp-dir\") pod \"simulator-main-045f69-kserve-75c7b5f8cf-wn9h4\" (UID: \"d1211d6c-1cd3-4b33-a2a1-28125b936f91\") " pod="e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4" Nov 01 06:55:27.735629 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.735426 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1211d6c-1cd3-4b33-a2a1-28125b936f91-dshm\") pod \"simulator-main-045f69-kserve-75c7b5f8cf-wn9h4\" (UID: \"d1211d6c-1cd3-4b33-a2a1-28125b936f91\") " pod="e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4" Nov 01 06:55:27.735629 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.735504 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4hln\" (UniqueName: \"kubernetes.io/projected/d1211d6c-1cd3-4b33-a2a1-28125b936f91-kube-api-access-w4hln\") pod \"simulator-main-045f69-kserve-75c7b5f8cf-wn9h4\" (UID: \"d1211d6c-1cd3-4b33-a2a1-28125b936f91\") " pod="e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4" Nov 01 06:55:27.735629 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.735535 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-main-045f69\"/\"kube-root-ca.crt\"" Nov 01 06:55:27.735629 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.735536 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-main-045f69\"/\"openshift-service-ca.crt\"" Nov 01 06:55:27.735629 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.735562 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1211d6c-1cd3-4b33-a2a1-28125b936f91-tls-certs\") pod \"simulator-main-045f69-kserve-75c7b5f8cf-wn9h4\" (UID: \"d1211d6c-1cd3-4b33-a2a1-28125b936f91\") " pod="e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4" Nov 01 06:55:27.735629 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.735583 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-main-045f69\"/\"simulator-main-045f69-kserve-self-signed-certs\"" Nov 01 06:55:27.735629 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.735609 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1211d6c-1cd3-4b33-a2a1-28125b936f91-home\") pod \"simulator-main-045f69-kserve-75c7b5f8cf-wn9h4\" (UID: \"d1211d6c-1cd3-4b33-a2a1-28125b936f91\") " pod="e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4" Nov 01 06:55:27.735629 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.735537 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-main-045f69\"/\"default-dockercfg-rtb5x\"" Nov 01 06:55:27.736058 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.735723 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1211d6c-1cd3-4b33-a2a1-28125b936f91-model-cache\") pod \"simulator-main-045f69-kserve-75c7b5f8cf-wn9h4\" (UID: \"d1211d6c-1cd3-4b33-a2a1-28125b936f91\") " pod="e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4" Nov 01 06:55:27.736058 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.735924 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1211d6c-1cd3-4b33-a2a1-28125b936f91-tmp-dir\") pod \"simulator-main-045f69-kserve-75c7b5f8cf-wn9h4\" (UID: \"d1211d6c-1cd3-4b33-a2a1-28125b936f91\") " pod="e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4" Nov 01 06:55:27.736058 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.735999 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1211d6c-1cd3-4b33-a2a1-28125b936f91-home\") pod \"simulator-main-045f69-kserve-75c7b5f8cf-wn9h4\" (UID: \"d1211d6c-1cd3-4b33-a2a1-28125b936f91\") " pod="e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4" Nov 01 06:55:27.736184 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.736074 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1211d6c-1cd3-4b33-a2a1-28125b936f91-model-cache\") pod \"simulator-main-045f69-kserve-75c7b5f8cf-wn9h4\" (UID: \"d1211d6c-1cd3-4b33-a2a1-28125b936f91\") " pod="e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4" Nov 01 06:55:27.737931 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.737907 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1211d6c-1cd3-4b33-a2a1-28125b936f91-dshm\") pod \"simulator-main-045f69-kserve-75c7b5f8cf-wn9h4\" (UID: \"d1211d6c-1cd3-4b33-a2a1-28125b936f91\") " pod="e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4" Nov 01 06:55:27.738093 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.738074 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1211d6c-1cd3-4b33-a2a1-28125b936f91-tls-certs\") pod \"simulator-main-045f69-kserve-75c7b5f8cf-wn9h4\" (UID: \"d1211d6c-1cd3-4b33-a2a1-28125b936f91\") " pod="e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4" Nov 01 06:55:27.797295 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.797260 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4hln\" (UniqueName: \"kubernetes.io/projected/d1211d6c-1cd3-4b33-a2a1-28125b936f91-kube-api-access-w4hln\") pod \"simulator-main-045f69-kserve-75c7b5f8cf-wn9h4\" (UID: \"d1211d6c-1cd3-4b33-a2a1-28125b936f91\") " pod="e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4" Nov 01 06:55:27.942820 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:27.942729 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4" Nov 01 06:55:28.318928 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:28.318900 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4"] Nov 01 06:55:28.320634 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:55:28.320606 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1211d6c_1cd3_4b33_a2a1_28125b936f91.slice/crio-c668c3a7b7a146436520a5a704567205d4d778f9f5dd181a2334097187eed0ca WatchSource:0}: Error finding container c668c3a7b7a146436520a5a704567205d4d778f9f5dd181a2334097187eed0ca: Status 404 returned error can't find the container with id c668c3a7b7a146436520a5a704567205d4d778f9f5dd181a2334097187eed0ca Nov 01 06:55:28.540897 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:28.540860 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4" event={"ID":"d1211d6c-1cd3-4b33-a2a1-28125b936f91","Type":"ContainerStarted","Data":"86457b1f501d41f7f950f01f010f3732bf7166f09c107d3e2e0bc7ce80e3343e"} Nov 01 06:55:28.541075 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:28.540907 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4" event={"ID":"d1211d6c-1cd3-4b33-a2a1-28125b936f91","Type":"ContainerStarted","Data":"c668c3a7b7a146436520a5a704567205d4d778f9f5dd181a2334097187eed0ca"} Nov 01 06:55:28.658322 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:28.658219 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4" podStartSLOduration=1.6582056870000002 podStartE2EDuration="1.658205687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:55:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:55:28.657613074 +0000 UTC m=+4207.312730241" watchObservedRunningTime="2026-11-01 06:55:28.658205687 +0000 UTC m=+4207.313322854" Nov 01 06:55:28.943072 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:28.942972 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4" Nov 01 06:55:32.047591 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:32.047551 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2dd2f7d-96s4m"] Nov 01 06:55:32.098153 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:32.098116 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2dd2f7d-96s4m"] Nov 01 06:55:33.045222 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:33.045183 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9856327a-ll857"] Nov 01 06:55:33.095579 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:33.095544 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9856327a-ll857"] Nov 01 06:55:33.514208 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:33.514174 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3460d306-df7a-4fcd-93ba-0bb4977553e5" path="/var/lib/kubelet/pods/3460d306-df7a-4fcd-93ba-0bb4977553e5/volumes" Nov 01 06:55:33.514721 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:33.514667 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f926ff2a-42d9-4af9-a597-c8734d9e1b06" path="/var/lib/kubelet/pods/f926ff2a-42d9-4af9-a597-c8734d9e1b06/volumes" Nov 01 06:55:37.943145 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:37.943108 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4" Nov 01 06:55:37.955474 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:37.955449 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4" Nov 01 06:55:38.610733 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:38.610701 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4" Nov 01 06:55:39.973082 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:55:39.973044 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:55:44.048978 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:44.048940 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b6dc894-pnghr"] Nov 01 06:55:44.099259 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:44.099222 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b6dc894-pnghr"] Nov 01 06:55:45.514066 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:45.514032 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07ee4cb6-922b-4fdd-b2b1-055c5302298a" path="/var/lib/kubelet/pods/07ee4cb6-922b-4fdd-b2b1-055c5302298a/volumes" Nov 01 06:55:46.046257 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:46.046223 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42ca33dc-wtmdf"] Nov 01 06:55:46.096846 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:46.096814 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42ca33dc-wtmdf"] Nov 01 06:55:47.514076 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:47.514039 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="464abd98-219f-4c01-a48c-7572b93f2cf6" path="/var/lib/kubelet/pods/464abd98-219f-4c01-a48c-7572b93f2cf6/volumes" Nov 01 06:55:54.794494 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:54.794448 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9"] Nov 01 06:55:54.794948 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:54.794830 5040 cpu_manager.go:414] "containerMap: removing container" podUID="932df59a-5bc9-4c43-a4ad-55dc8b516b7b" containerName="istio-proxy" Nov 01 06:55:54.794948 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:54.794842 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="932df59a-5bc9-4c43-a4ad-55dc8b516b7b" containerName="istio-proxy" Nov 01 06:55:54.794948 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:54.794878 5040 cpu_manager.go:414] "containerMap: removing container" podUID="14b4da92-5164-45df-8306-5f228a7c5ac2" containerName="istio-proxy" Nov 01 06:55:54.794948 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:54.794883 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14b4da92-5164-45df-8306-5f228a7c5ac2" containerName="istio-proxy" Nov 01 06:55:54.794948 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:54.794943 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="14b4da92-5164-45df-8306-5f228a7c5ac2" containerName="istio-proxy" Nov 01 06:55:54.799388 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:54.799368 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9" Nov 01 06:55:54.862123 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:54.862088 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/351df948-ac15-469b-a606-c84b4b9cf3c9-dshm\") pod \"premium-main-045f69-kserve-588987bfc4-hfpz9\" (UID: \"351df948-ac15-469b-a606-c84b4b9cf3c9\") " pod="e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9" Nov 01 06:55:54.862292 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:54.862149 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/351df948-ac15-469b-a606-c84b4b9cf3c9-home\") pod \"premium-main-045f69-kserve-588987bfc4-hfpz9\" (UID: \"351df948-ac15-469b-a606-c84b4b9cf3c9\") " pod="e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9" Nov 01 06:55:54.862292 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:54.862218 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lllbv\" (UniqueName: \"kubernetes.io/projected/351df948-ac15-469b-a606-c84b4b9cf3c9-kube-api-access-lllbv\") pod \"premium-main-045f69-kserve-588987bfc4-hfpz9\" (UID: \"351df948-ac15-469b-a606-c84b4b9cf3c9\") " pod="e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9" Nov 01 06:55:54.862292 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:54.862253 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/351df948-ac15-469b-a606-c84b4b9cf3c9-tls-certs\") pod \"premium-main-045f69-kserve-588987bfc4-hfpz9\" (UID: \"351df948-ac15-469b-a606-c84b4b9cf3c9\") " pod="e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9" Nov 01 06:55:54.862499 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:54.862293 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/351df948-ac15-469b-a606-c84b4b9cf3c9-tmp-dir\") pod \"premium-main-045f69-kserve-588987bfc4-hfpz9\" (UID: \"351df948-ac15-469b-a606-c84b4b9cf3c9\") " pod="e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9" Nov 01 06:55:54.862499 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:54.862387 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/351df948-ac15-469b-a606-c84b4b9cf3c9-model-cache\") pod \"premium-main-045f69-kserve-588987bfc4-hfpz9\" (UID: \"351df948-ac15-469b-a606-c84b4b9cf3c9\") " pod="e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9" Nov 01 06:55:54.904926 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:54.904894 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-main-045f69\"/\"premium-main-045f69-kserve-self-signed-certs\"" Nov 01 06:55:54.907095 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:54.907074 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9"] Nov 01 06:55:54.963284 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:54.963246 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/351df948-ac15-469b-a606-c84b4b9cf3c9-model-cache\") pod \"premium-main-045f69-kserve-588987bfc4-hfpz9\" (UID: \"351df948-ac15-469b-a606-c84b4b9cf3c9\") " pod="e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9" Nov 01 06:55:54.963557 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:54.963312 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/351df948-ac15-469b-a606-c84b4b9cf3c9-dshm\") pod \"premium-main-045f69-kserve-588987bfc4-hfpz9\" (UID: \"351df948-ac15-469b-a606-c84b4b9cf3c9\") " pod="e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9" Nov 01 06:55:54.963557 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:54.963379 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/351df948-ac15-469b-a606-c84b4b9cf3c9-home\") pod \"premium-main-045f69-kserve-588987bfc4-hfpz9\" (UID: \"351df948-ac15-469b-a606-c84b4b9cf3c9\") " pod="e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9" Nov 01 06:55:54.963557 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:54.963409 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lllbv\" (UniqueName: \"kubernetes.io/projected/351df948-ac15-469b-a606-c84b4b9cf3c9-kube-api-access-lllbv\") pod \"premium-main-045f69-kserve-588987bfc4-hfpz9\" (UID: \"351df948-ac15-469b-a606-c84b4b9cf3c9\") " pod="e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9" Nov 01 06:55:54.963557 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:54.963440 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/351df948-ac15-469b-a606-c84b4b9cf3c9-tls-certs\") pod \"premium-main-045f69-kserve-588987bfc4-hfpz9\" (UID: \"351df948-ac15-469b-a606-c84b4b9cf3c9\") " pod="e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9" Nov 01 06:55:54.963557 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:54.963508 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/351df948-ac15-469b-a606-c84b4b9cf3c9-tmp-dir\") pod \"premium-main-045f69-kserve-588987bfc4-hfpz9\" (UID: \"351df948-ac15-469b-a606-c84b4b9cf3c9\") " pod="e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9" Nov 01 06:55:54.963853 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:54.963785 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/351df948-ac15-469b-a606-c84b4b9cf3c9-model-cache\") pod \"premium-main-045f69-kserve-588987bfc4-hfpz9\" (UID: \"351df948-ac15-469b-a606-c84b4b9cf3c9\") " pod="e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9" Nov 01 06:55:54.963853 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:54.963830 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/351df948-ac15-469b-a606-c84b4b9cf3c9-home\") pod \"premium-main-045f69-kserve-588987bfc4-hfpz9\" (UID: \"351df948-ac15-469b-a606-c84b4b9cf3c9\") " pod="e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9" Nov 01 06:55:54.963926 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:54.963864 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/351df948-ac15-469b-a606-c84b4b9cf3c9-tmp-dir\") pod \"premium-main-045f69-kserve-588987bfc4-hfpz9\" (UID: \"351df948-ac15-469b-a606-c84b4b9cf3c9\") " pod="e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9" Nov 01 06:55:54.965736 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:54.965717 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/351df948-ac15-469b-a606-c84b4b9cf3c9-dshm\") pod \"premium-main-045f69-kserve-588987bfc4-hfpz9\" (UID: \"351df948-ac15-469b-a606-c84b4b9cf3c9\") " pod="e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9" Nov 01 06:55:54.966013 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:54.965992 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/351df948-ac15-469b-a606-c84b4b9cf3c9-tls-certs\") pod \"premium-main-045f69-kserve-588987bfc4-hfpz9\" (UID: \"351df948-ac15-469b-a606-c84b4b9cf3c9\") " pod="e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9" Nov 01 06:55:55.027098 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:55.027058 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lllbv\" (UniqueName: \"kubernetes.io/projected/351df948-ac15-469b-a606-c84b4b9cf3c9-kube-api-access-lllbv\") pod \"premium-main-045f69-kserve-588987bfc4-hfpz9\" (UID: \"351df948-ac15-469b-a606-c84b4b9cf3c9\") " pod="e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9" Nov 01 06:55:55.111654 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:55.111558 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9" Nov 01 06:55:55.486866 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:55.486837 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9"] Nov 01 06:55:55.488461 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:55:55.488430 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod351df948_ac15_469b_a606_c84b4b9cf3c9.slice/crio-aaf81d54cc8753f44fe84d350b8acbd1d49812a899119e85a0ec7b39b290fbdf WatchSource:0}: Error finding container aaf81d54cc8753f44fe84d350b8acbd1d49812a899119e85a0ec7b39b290fbdf: Status 404 returned error can't find the container with id aaf81d54cc8753f44fe84d350b8acbd1d49812a899119e85a0ec7b39b290fbdf Nov 01 06:55:55.691776 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:55.691729 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9" event={"ID":"351df948-ac15-469b-a606-c84b4b9cf3c9","Type":"ContainerStarted","Data":"f9bd70072f90866697ff14de71ec75b51a62b0cd955921c0d3fd29510da59d46"} Nov 01 06:55:55.691776 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:55.691776 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9" event={"ID":"351df948-ac15-469b-a606-c84b4b9cf3c9","Type":"ContainerStarted","Data":"aaf81d54cc8753f44fe84d350b8acbd1d49812a899119e85a0ec7b39b290fbdf"} Nov 01 06:55:55.811383 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:55.811330 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9" podStartSLOduration=1.8113145830000001 podStartE2EDuration="1.811314583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:55:55.809417182 +0000 UTC m=+4234.464534352" watchObservedRunningTime="2026-11-01 06:55:55.811314583 +0000 UTC m=+4234.466431750" Nov 01 06:55:56.048748 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:56.048709 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29804d8b-zjzkl"] Nov 01 06:55:56.051256 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:56.051234 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29804d8b-zjzkl"] Nov 01 06:55:56.112525 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:56.112434 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9" Nov 01 06:55:57.514003 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:55:57.513968 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3381c8ac-63b7-4df0-b3f5-88579e89489e" path="/var/lib/kubelet/pods/3381c8ac-63b7-4df0-b3f5-88579e89489e/volumes" Nov 01 06:56:05.112672 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:56:05.112620 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9" Nov 01 06:56:05.125496 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:56:05.125444 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9" Nov 01 06:56:05.760237 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:56:05.760208 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9" Nov 01 06:56:09.971832 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:56:09.971797 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:56:24.251805 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:56:24.251771 5040 scope.go:122] "RemoveContainer" containerID="9f84efb1a2bf597369e371f5f8217ee8dcea1d8ce387bba4fe1bf66772e2d449" Nov 01 06:56:24.260763 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:56:24.260738 5040 scope.go:122] "RemoveContainer" containerID="428e8d09f6b4887f7222e4d534df73a9327106879f049cb76676ccf9f64233f9" Nov 01 06:56:39.972248 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:56:39.972166 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:57:09.174744 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:09.174702 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b"] Nov 01 06:57:09.178262 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:09.178244 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" Nov 01 06:57:09.231195 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:09.231162 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 01 06:57:09.231384 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:09.231358 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-vqwlh\"" Nov 01 06:57:09.245139 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:09.245104 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b85392e-5dd8-4539-93ac-e659fb9cdf2c-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-f895b\" (UID: \"2b85392e-5dd8-4539-93ac-e659fb9cdf2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" Nov 01 06:57:09.245273 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:09.245167 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b85392e-5dd8-4539-93ac-e659fb9cdf2c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-f895b\" (UID: \"2b85392e-5dd8-4539-93ac-e659fb9cdf2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" Nov 01 06:57:09.245273 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:09.245246 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b85392e-5dd8-4539-93ac-e659fb9cdf2c-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-f895b\" (UID: \"2b85392e-5dd8-4539-93ac-e659fb9cdf2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" Nov 01 06:57:09.245382 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:09.245286 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b85392e-5dd8-4539-93ac-e659fb9cdf2c-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-f895b\" (UID: \"2b85392e-5dd8-4539-93ac-e659fb9cdf2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" Nov 01 06:57:09.245382 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:09.245310 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b85392e-5dd8-4539-93ac-e659fb9cdf2c-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-f895b\" (UID: \"2b85392e-5dd8-4539-93ac-e659fb9cdf2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" Nov 01 06:57:09.245463 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:09.245430 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b85392e-5dd8-4539-93ac-e659fb9cdf2c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-f895b\" (UID: \"2b85392e-5dd8-4539-93ac-e659fb9cdf2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" Nov 01 06:57:09.245550 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:09.245464 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b85392e-5dd8-4539-93ac-e659fb9cdf2c-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-f895b\" (UID: \"2b85392e-5dd8-4539-93ac-e659fb9cdf2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" Nov 01 06:57:09.245597 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:09.245571 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b85392e-5dd8-4539-93ac-e659fb9cdf2c-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-f895b\" (UID: \"2b85392e-5dd8-4539-93ac-e659fb9cdf2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" Nov 01 06:57:09.245597 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:09.245592 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnv7c\" (UniqueName: \"kubernetes.io/projected/2b85392e-5dd8-4539-93ac-e659fb9cdf2c-kube-api-access-tnv7c\") pod \"maas-default-gateway-openshift-default-855f9574cf-f895b\" (UID: \"2b85392e-5dd8-4539-93ac-e659fb9cdf2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" Nov 01 06:57:09.284657 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:09.284622 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b"] Nov 01 06:57:09.346310 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:09.346271 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b85392e-5dd8-4539-93ac-e659fb9cdf2c-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-f895b\" (UID: \"2b85392e-5dd8-4539-93ac-e659fb9cdf2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" Nov 01 06:57:09.346539 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:09.346351 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b85392e-5dd8-4539-93ac-e659fb9cdf2c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-f895b\" (UID: \"2b85392e-5dd8-4539-93ac-e659fb9cdf2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" Nov 01 06:57:09.346539 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:09.346386 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b85392e-5dd8-4539-93ac-e659fb9cdf2c-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-f895b\" (UID: \"2b85392e-5dd8-4539-93ac-e659fb9cdf2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" Nov 01 06:57:09.346539 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:09.346519 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b85392e-5dd8-4539-93ac-e659fb9cdf2c-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-f895b\" (UID: \"2b85392e-5dd8-4539-93ac-e659fb9cdf2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" Nov 01 06:57:09.346750 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:09.346563 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b85392e-5dd8-4539-93ac-e659fb9cdf2c-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-f895b\" (UID: \"2b85392e-5dd8-4539-93ac-e659fb9cdf2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" Nov 01 06:57:09.346750 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:09.346598 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b85392e-5dd8-4539-93ac-e659fb9cdf2c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-f895b\" (UID: \"2b85392e-5dd8-4539-93ac-e659fb9cdf2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" Nov 01 06:57:09.346750 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:09.346635 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b85392e-5dd8-4539-93ac-e659fb9cdf2c-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-f895b\" (UID: \"2b85392e-5dd8-4539-93ac-e659fb9cdf2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" Nov 01 06:57:09.346750 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:09.346679 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b85392e-5dd8-4539-93ac-e659fb9cdf2c-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-f895b\" (UID: \"2b85392e-5dd8-4539-93ac-e659fb9cdf2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" Nov 01 06:57:09.346750 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:09.346686 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b85392e-5dd8-4539-93ac-e659fb9cdf2c-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-f895b\" (UID: \"2b85392e-5dd8-4539-93ac-e659fb9cdf2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" Nov 01 06:57:09.346750 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:09.346703 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnv7c\" (UniqueName: \"kubernetes.io/projected/2b85392e-5dd8-4539-93ac-e659fb9cdf2c-kube-api-access-tnv7c\") pod \"maas-default-gateway-openshift-default-855f9574cf-f895b\" (UID: \"2b85392e-5dd8-4539-93ac-e659fb9cdf2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" Nov 01 06:57:09.347123 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:09.346757 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b85392e-5dd8-4539-93ac-e659fb9cdf2c-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-f895b\" (UID: \"2b85392e-5dd8-4539-93ac-e659fb9cdf2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" Nov 01 06:57:09.347123 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:09.346958 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b85392e-5dd8-4539-93ac-e659fb9cdf2c-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-f895b\" (UID: \"2b85392e-5dd8-4539-93ac-e659fb9cdf2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" Nov 01 06:57:09.347123 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:09.347113 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b85392e-5dd8-4539-93ac-e659fb9cdf2c-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-f895b\" (UID: \"2b85392e-5dd8-4539-93ac-e659fb9cdf2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" Nov 01 06:57:09.347238 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:09.347135 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b85392e-5dd8-4539-93ac-e659fb9cdf2c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-f895b\" (UID: \"2b85392e-5dd8-4539-93ac-e659fb9cdf2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" Nov 01 06:57:09.348865 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:09.348843 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b85392e-5dd8-4539-93ac-e659fb9cdf2c-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-f895b\" (UID: \"2b85392e-5dd8-4539-93ac-e659fb9cdf2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" Nov 01 06:57:09.349186 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:09.349167 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b85392e-5dd8-4539-93ac-e659fb9cdf2c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-f895b\" (UID: \"2b85392e-5dd8-4539-93ac-e659fb9cdf2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" Nov 01 06:57:09.408373 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:09.408348 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b85392e-5dd8-4539-93ac-e659fb9cdf2c-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-f895b\" (UID: \"2b85392e-5dd8-4539-93ac-e659fb9cdf2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" Nov 01 06:57:09.408548 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:09.408440 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnv7c\" (UniqueName: \"kubernetes.io/projected/2b85392e-5dd8-4539-93ac-e659fb9cdf2c-kube-api-access-tnv7c\") pod \"maas-default-gateway-openshift-default-855f9574cf-f895b\" (UID: \"2b85392e-5dd8-4539-93ac-e659fb9cdf2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" Nov 01 06:57:09.490614 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:09.490548 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" Nov 01 06:57:09.916634 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:09.916602 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b"] Nov 01 06:57:09.918583 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:57:09.918553 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b85392e_5dd8_4539_93ac_e659fb9cdf2c.slice/crio-c7e97d30fd84d58366dbc6d0f8c044e7d23ff30ddea77cc1fbe18e24def5c14a WatchSource:0}: Error finding container c7e97d30fd84d58366dbc6d0f8c044e7d23ff30ddea77cc1fbe18e24def5c14a: Status 404 returned error can't find the container with id c7e97d30fd84d58366dbc6d0f8c044e7d23ff30ddea77cc1fbe18e24def5c14a Nov 01 06:57:09.920638 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:09.920602 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:57:09.920707 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:09.920668 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:57:09.920707 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:09.920697 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:57:09.971803 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:57:09.971767 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:57:10.106508 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:10.106460 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" event={"ID":"2b85392e-5dd8-4539-93ac-e659fb9cdf2c","Type":"ContainerStarted","Data":"81d4ef92de7fea19fe796d8fea4fca9dab02a357f6b6405622b4a0ce09975ee7"} Nov 01 06:57:10.106701 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:10.106532 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" event={"ID":"2b85392e-5dd8-4539-93ac-e659fb9cdf2c","Type":"ContainerStarted","Data":"c7e97d30fd84d58366dbc6d0f8c044e7d23ff30ddea77cc1fbe18e24def5c14a"} Nov 01 06:57:10.229904 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:10.229793 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" podStartSLOduration=1.229778599 podStartE2EDuration="1.229778599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:57:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:57:10.226835759 +0000 UTC m=+4308.881952928" watchObservedRunningTime="2026-11-01 06:57:10.229778599 +0000 UTC m=+4308.884895766" Nov 01 06:57:10.491429 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:10.491338 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" Nov 01 06:57:10.493105 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:10.493076 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-f895b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body= Nov 01 06:57:10.493225 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:10.493138 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" podUID="2b85392e-5dd8-4539-93ac-e659fb9cdf2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" Nov 01 06:57:11.490915 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:11.490883 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-f895b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body= Nov 01 06:57:11.491280 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:11.490937 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" podUID="2b85392e-5dd8-4539-93ac-e659fb9cdf2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" Nov 01 06:57:12.491663 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:12.491624 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-f895b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body= Nov 01 06:57:12.492072 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:12.491696 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" podUID="2b85392e-5dd8-4539-93ac-e659fb9cdf2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" Nov 01 06:57:13.491610 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:13.491570 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-f895b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body= Nov 01 06:57:13.491835 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:13.491626 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" podUID="2b85392e-5dd8-4539-93ac-e659fb9cdf2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" Nov 01 06:57:14.491746 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:14.491705 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-f895b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body= Nov 01 06:57:14.491950 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:14.491762 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" podUID="2b85392e-5dd8-4539-93ac-e659fb9cdf2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" Nov 01 06:57:15.491821 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:15.491783 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-f895b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body= Nov 01 06:57:15.492096 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:15.491833 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" podUID="2b85392e-5dd8-4539-93ac-e659fb9cdf2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" Nov 01 06:57:16.491789 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:16.491744 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-f895b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body= Nov 01 06:57:16.491986 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:16.491821 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" podUID="2b85392e-5dd8-4539-93ac-e659fb9cdf2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" Nov 01 06:57:17.490898 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:17.490862 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-f895b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body= Nov 01 06:57:17.491319 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:17.490916 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" podUID="2b85392e-5dd8-4539-93ac-e659fb9cdf2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" Nov 01 06:57:19.532698 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:19.532651 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-f895b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 06:57:19.533132 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:19.532740 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" podUID="2b85392e-5dd8-4539-93ac-e659fb9cdf2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 06:57:20.349022 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:20.348987 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" Nov 01 06:57:20.349425 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:20.349390 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" Nov 01 06:57:20.350621 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:20.350508 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" Nov 01 06:57:24.180319 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.180280 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d"] Nov 01 06:57:24.184246 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.184222 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" Nov 01 06:57:24.287175 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.287141 5040 scope.go:122] "RemoveContainer" containerID="de43637c2774a1a8cd45c84f2f296135b05ab0928fdf8d45e88a71e3a3769e60" Nov 01 06:57:24.296290 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.296263 5040 scope.go:122] "RemoveContainer" containerID="6cef364a1fa909d3c7a57e861d028cc604510aa6cb54e2450322f8c1300a0b6e" Nov 01 06:57:24.303351 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.303323 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fea2872-e620-4756-9ede-22f65d64cf03-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-jl24d\" (UID: \"4fea2872-e620-4756-9ede-22f65d64cf03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" Nov 01 06:57:24.303493 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.303367 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fea2872-e620-4756-9ede-22f65d64cf03-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-jl24d\" (UID: \"4fea2872-e620-4756-9ede-22f65d64cf03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" Nov 01 06:57:24.303493 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.303460 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fea2872-e620-4756-9ede-22f65d64cf03-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-jl24d\" (UID: \"4fea2872-e620-4756-9ede-22f65d64cf03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" Nov 01 06:57:24.303590 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.303549 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fea2872-e620-4756-9ede-22f65d64cf03-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-jl24d\" (UID: \"4fea2872-e620-4756-9ede-22f65d64cf03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" Nov 01 06:57:24.303590 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.303582 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fea2872-e620-4756-9ede-22f65d64cf03-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-jl24d\" (UID: \"4fea2872-e620-4756-9ede-22f65d64cf03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" Nov 01 06:57:24.303677 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.303605 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fea2872-e620-4756-9ede-22f65d64cf03-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-jl24d\" (UID: \"4fea2872-e620-4756-9ede-22f65d64cf03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" Nov 01 06:57:24.303728 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.303704 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fea2872-e620-4756-9ede-22f65d64cf03-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-jl24d\" (UID: \"4fea2872-e620-4756-9ede-22f65d64cf03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" Nov 01 06:57:24.303771 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.303742 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fea2872-e620-4756-9ede-22f65d64cf03-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-jl24d\" (UID: \"4fea2872-e620-4756-9ede-22f65d64cf03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" Nov 01 06:57:24.303857 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.303841 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp6qh\" (UniqueName: \"kubernetes.io/projected/4fea2872-e620-4756-9ede-22f65d64cf03-kube-api-access-jp6qh\") pod \"maas-default-gateway-openshift-default-855f9574cf-jl24d\" (UID: \"4fea2872-e620-4756-9ede-22f65d64cf03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" Nov 01 06:57:24.304805 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.304787 5040 scope.go:122] "RemoveContainer" containerID="77b73c7a19753cd1fba3d4ef4e4938098aaecbc2a1ab700ccf2eecf3873c91ad" Nov 01 06:57:24.312603 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.312575 5040 scope.go:122] "RemoveContainer" containerID="ec00f70688314b481a4fdd17328bf4357ee0a8d615a62b9737ff426c372b90aa" Nov 01 06:57:24.320250 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.320234 5040 scope.go:122] "RemoveContainer" containerID="4dca403bcbed3a8a774dcfa3a27df021985b95355fb3480b60e4fee2c52efdaa" Nov 01 06:57:24.391944 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.391912 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d"] Nov 01 06:57:24.405288 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.405262 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fea2872-e620-4756-9ede-22f65d64cf03-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-jl24d\" (UID: \"4fea2872-e620-4756-9ede-22f65d64cf03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" Nov 01 06:57:24.405419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.405300 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fea2872-e620-4756-9ede-22f65d64cf03-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-jl24d\" (UID: \"4fea2872-e620-4756-9ede-22f65d64cf03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" Nov 01 06:57:24.405419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.405348 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fea2872-e620-4756-9ede-22f65d64cf03-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-jl24d\" (UID: \"4fea2872-e620-4756-9ede-22f65d64cf03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" Nov 01 06:57:24.405419 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.405384 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fea2872-e620-4756-9ede-22f65d64cf03-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-jl24d\" (UID: \"4fea2872-e620-4756-9ede-22f65d64cf03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" Nov 01 06:57:24.405553 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.405524 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fea2872-e620-4756-9ede-22f65d64cf03-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-jl24d\" (UID: \"4fea2872-e620-4756-9ede-22f65d64cf03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" Nov 01 06:57:24.405628 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.405611 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp6qh\" (UniqueName: \"kubernetes.io/projected/4fea2872-e620-4756-9ede-22f65d64cf03-kube-api-access-jp6qh\") pod \"maas-default-gateway-openshift-default-855f9574cf-jl24d\" (UID: \"4fea2872-e620-4756-9ede-22f65d64cf03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" Nov 01 06:57:24.405684 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.405664 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fea2872-e620-4756-9ede-22f65d64cf03-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-jl24d\" (UID: \"4fea2872-e620-4756-9ede-22f65d64cf03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" Nov 01 06:57:24.405749 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.405700 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fea2872-e620-4756-9ede-22f65d64cf03-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-jl24d\" (UID: \"4fea2872-e620-4756-9ede-22f65d64cf03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" Nov 01 06:57:24.405749 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.405717 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fea2872-e620-4756-9ede-22f65d64cf03-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-jl24d\" (UID: \"4fea2872-e620-4756-9ede-22f65d64cf03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" Nov 01 06:57:24.405749 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.405737 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fea2872-e620-4756-9ede-22f65d64cf03-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-jl24d\" (UID: \"4fea2872-e620-4756-9ede-22f65d64cf03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" Nov 01 06:57:24.405916 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.405793 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fea2872-e620-4756-9ede-22f65d64cf03-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-jl24d\" (UID: \"4fea2872-e620-4756-9ede-22f65d64cf03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" Nov 01 06:57:24.406078 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.406057 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fea2872-e620-4756-9ede-22f65d64cf03-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-jl24d\" (UID: \"4fea2872-e620-4756-9ede-22f65d64cf03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" Nov 01 06:57:24.406277 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.406244 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fea2872-e620-4756-9ede-22f65d64cf03-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-jl24d\" (UID: \"4fea2872-e620-4756-9ede-22f65d64cf03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" Nov 01 06:57:24.406342 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.406280 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fea2872-e620-4756-9ede-22f65d64cf03-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-jl24d\" (UID: \"4fea2872-e620-4756-9ede-22f65d64cf03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" Nov 01 06:57:24.408014 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.407993 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fea2872-e620-4756-9ede-22f65d64cf03-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-jl24d\" (UID: \"4fea2872-e620-4756-9ede-22f65d64cf03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" Nov 01 06:57:24.408260 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.408241 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fea2872-e620-4756-9ede-22f65d64cf03-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-jl24d\" (UID: \"4fea2872-e620-4756-9ede-22f65d64cf03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" Nov 01 06:57:24.467386 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.467314 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fea2872-e620-4756-9ede-22f65d64cf03-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-jl24d\" (UID: \"4fea2872-e620-4756-9ede-22f65d64cf03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" Nov 01 06:57:24.467386 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.467343 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp6qh\" (UniqueName: \"kubernetes.io/projected/4fea2872-e620-4756-9ede-22f65d64cf03-kube-api-access-jp6qh\") pod \"maas-default-gateway-openshift-default-855f9574cf-jl24d\" (UID: \"4fea2872-e620-4756-9ede-22f65d64cf03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" Nov 01 06:57:24.497528 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.497471 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" Nov 01 06:57:24.876420 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.876380 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d"] Nov 01 06:57:24.913104 ip-10-0-143-195 kubenswrapper[5040]: W1101 06:57:24.913065 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fea2872_e620_4756_9ede_22f65d64cf03.slice/crio-f5bfb1905b024c41cefc9fe8985f812f1ef012508aa99e2e419e94a7b8c50d51 WatchSource:0}: Error finding container f5bfb1905b024c41cefc9fe8985f812f1ef012508aa99e2e419e94a7b8c50d51: Status 404 returned error can't find the container with id f5bfb1905b024c41cefc9fe8985f812f1ef012508aa99e2e419e94a7b8c50d51 Nov 01 06:57:24.915315 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.915276 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:57:24.915406 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.915350 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:57:24.915406 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:24.915379 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 06:57:25.205892 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:25.205781 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" event={"ID":"4fea2872-e620-4756-9ede-22f65d64cf03","Type":"ContainerStarted","Data":"b88eac6c4c001036745fb374e9f2bb06c3bebc60b5883ac5861db33789ff848a"} Nov 01 06:57:25.205892 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:25.205857 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" event={"ID":"4fea2872-e620-4756-9ede-22f65d64cf03","Type":"ContainerStarted","Data":"f5bfb1905b024c41cefc9fe8985f812f1ef012508aa99e2e419e94a7b8c50d51"} Nov 01 06:57:25.329714 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:25.329660 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" podStartSLOduration=1.32964605 podStartE2EDuration="1.32964605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:57:25.326885223 +0000 UTC m=+4323.982002390" watchObservedRunningTime="2026-11-01 06:57:25.32964605 +0000 UTC m=+4323.984763219" Nov 01 06:57:25.520782 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:25.520744 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-jl24d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Nov 01 06:57:25.520984 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:25.520810 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" podUID="4fea2872-e620-4756-9ede-22f65d64cf03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Nov 01 06:57:25.524276 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:25.524250 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" Nov 01 06:57:26.514504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:26.503576 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-jl24d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Nov 01 06:57:26.514504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:26.503652 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" podUID="4fea2872-e620-4756-9ede-22f65d64cf03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Nov 01 06:57:27.506504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:27.501875 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-jl24d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Nov 01 06:57:27.506504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:27.501947 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" podUID="4fea2872-e620-4756-9ede-22f65d64cf03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Nov 01 06:57:28.507505 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:28.497860 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-jl24d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Nov 01 06:57:28.507505 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:28.497924 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" podUID="4fea2872-e620-4756-9ede-22f65d64cf03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Nov 01 06:57:29.511512 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:29.511332 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-jl24d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Nov 01 06:57:29.511512 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:29.511395 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" podUID="4fea2872-e620-4756-9ede-22f65d64cf03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Nov 01 06:57:30.504504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:30.497824 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-jl24d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Nov 01 06:57:30.504504 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:30.497885 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" podUID="4fea2872-e620-4756-9ede-22f65d64cf03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Nov 01 06:57:31.498806 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:31.498770 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-jl24d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Nov 01 06:57:31.499188 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:31.498825 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" podUID="4fea2872-e620-4756-9ede-22f65d64cf03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Nov 01 06:57:33.113358 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:33.113325 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" Nov 01 06:57:33.113891 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:33.113806 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" Nov 01 06:57:33.115065 ip-10-0-143-195 kubenswrapper[5040]: I1101 06:57:33.115042 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" Nov 01 06:57:39.972060 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:57:39.972022 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:58:09.972413 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:58:09.972377 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:58:39.972894 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:58:39.972855 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:59:09.971996 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:59:09.971958 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 06:59:39.972725 ip-10-0-143-195 kubenswrapper[5040]: E1101 06:59:39.972646 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:00:00.166259 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:00.166214 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-045f69-29891940-jtpxg"] Nov 01 07:00:00.170000 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:00.169977 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-045f69-29891940-jtpxg" Nov 01 07:00:00.223837 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:00.223803 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 07:00:00.226293 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:00.226264 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8daa9ba3-0fb1-4614-9013-8c907d18b85c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-main-045f69-29891940-jtpxg\" (UID: \"8daa9ba3-0fb1-4614-9013-8c907d18b85c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-045f69-29891940-jtpxg" Nov 01 07:00:00.226429 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:00.226402 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp45k\" (UniqueName: \"kubernetes.io/projected/8daa9ba3-0fb1-4614-9013-8c907d18b85c-kube-api-access-tp45k\") pod \"maas-api-key-cleanup-e2e-worker-main-045f69-29891940-jtpxg\" (UID: \"8daa9ba3-0fb1-4614-9013-8c907d18b85c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-045f69-29891940-jtpxg" Nov 01 07:00:00.274661 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:00.274631 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 07:00:00.274661 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:00.274636 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\"" Nov 01 07:00:00.277086 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:00.277065 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-045f69-29891940-jtpxg"] Nov 01 07:00:00.327338 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:00.327303 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8daa9ba3-0fb1-4614-9013-8c907d18b85c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-main-045f69-29891940-jtpxg\" (UID: \"8daa9ba3-0fb1-4614-9013-8c907d18b85c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-045f69-29891940-jtpxg" Nov 01 07:00:00.327561 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:00.327358 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp45k\" (UniqueName: \"kubernetes.io/projected/8daa9ba3-0fb1-4614-9013-8c907d18b85c-kube-api-access-tp45k\") pod \"maas-api-key-cleanup-e2e-worker-main-045f69-29891940-jtpxg\" (UID: \"8daa9ba3-0fb1-4614-9013-8c907d18b85c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-045f69-29891940-jtpxg" Nov 01 07:00:00.328086 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:00.328062 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8daa9ba3-0fb1-4614-9013-8c907d18b85c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-main-045f69-29891940-jtpxg\" (UID: \"8daa9ba3-0fb1-4614-9013-8c907d18b85c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-045f69-29891940-jtpxg" Nov 01 07:00:00.389277 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:00.389244 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp45k\" (UniqueName: \"kubernetes.io/projected/8daa9ba3-0fb1-4614-9013-8c907d18b85c-kube-api-access-tp45k\") pod \"maas-api-key-cleanup-e2e-worker-main-045f69-29891940-jtpxg\" (UID: \"8daa9ba3-0fb1-4614-9013-8c907d18b85c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-045f69-29891940-jtpxg" Nov 01 07:00:00.482933 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:00.482840 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-045f69-29891940-jtpxg" Nov 01 07:00:00.862458 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:00.862424 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-045f69-29891940-jtpxg"] Nov 01 07:00:00.864125 ip-10-0-143-195 kubenswrapper[5040]: W1101 07:00:00.864098 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8daa9ba3_0fb1_4614_9013_8c907d18b85c.slice/crio-d19c47b672fca620f464f4bdd82164f37ddfffe6cbad18630f1ba0de78a36b1e WatchSource:0}: Error finding container d19c47b672fca620f464f4bdd82164f37ddfffe6cbad18630f1ba0de78a36b1e: Status 404 returned error can't find the container with id d19c47b672fca620f464f4bdd82164f37ddfffe6cbad18630f1ba0de78a36b1e Nov 01 07:00:01.047614 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:01.047576 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-045f69-29891940-jtpxg" event={"ID":"8daa9ba3-0fb1-4614-9013-8c907d18b85c","Type":"ContainerStarted","Data":"d19c47b672fca620f464f4bdd82164f37ddfffe6cbad18630f1ba0de78a36b1e"} Nov 01 07:00:02.054360 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:02.054324 5040 generic.go:356] "Generic (PLEG): container finished" podID="8daa9ba3-0fb1-4614-9013-8c907d18b85c" containerID="a0d8bf216842248ed08e671347c4b2a76c1874c87e9d7328a842a5325303cef3" exitCode=60 Nov 01 07:00:02.054831 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:02.054407 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-045f69-29891940-jtpxg" event={"ID":"8daa9ba3-0fb1-4614-9013-8c907d18b85c","Type":"ContainerDied","Data":"a0d8bf216842248ed08e671347c4b2a76c1874c87e9d7328a842a5325303cef3"} Nov 01 07:00:02.054831 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:02.054682 5040 scope.go:122] "RemoveContainer" containerID="a0d8bf216842248ed08e671347c4b2a76c1874c87e9d7328a842a5325303cef3" Nov 01 07:00:03.062415 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:03.062322 5040 generic.go:356] "Generic (PLEG): container finished" podID="8daa9ba3-0fb1-4614-9013-8c907d18b85c" containerID="3d8712cbeaf692c9d1c185a9ef1d1d0a5e954b762e1ba9f9a6ee923949a19020" exitCode=60 Nov 01 07:00:03.062415 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:03.062401 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-045f69-29891940-jtpxg" event={"ID":"8daa9ba3-0fb1-4614-9013-8c907d18b85c","Type":"ContainerDied","Data":"3d8712cbeaf692c9d1c185a9ef1d1d0a5e954b762e1ba9f9a6ee923949a19020"} Nov 01 07:00:03.062889 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:03.062458 5040 scope.go:122] "RemoveContainer" containerID="a0d8bf216842248ed08e671347c4b2a76c1874c87e9d7328a842a5325303cef3" Nov 01 07:00:03.062889 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:03.062710 5040 scope.go:122] "RemoveContainer" containerID="3d8712cbeaf692c9d1c185a9ef1d1d0a5e954b762e1ba9f9a6ee923949a19020" Nov 01 07:00:03.062999 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:00:03.062978 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-main-045f69-29891940-jtpxg_odh-ai-gateway-infra(8daa9ba3-0fb1-4614-9013-8c907d18b85c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-045f69-29891940-jtpxg" podUID="8daa9ba3-0fb1-4614-9013-8c907d18b85c" Nov 01 07:00:04.069391 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:04.069357 5040 scope.go:122] "RemoveContainer" containerID="3d8712cbeaf692c9d1c185a9ef1d1d0a5e954b762e1ba9f9a6ee923949a19020" Nov 01 07:00:04.069881 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:00:04.069665 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-main-045f69-29891940-jtpxg_odh-ai-gateway-infra(8daa9ba3-0fb1-4614-9013-8c907d18b85c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-045f69-29891940-jtpxg" podUID="8daa9ba3-0fb1-4614-9013-8c907d18b85c" Nov 01 07:00:09.972386 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:00:09.972345 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:00:15.507999 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:15.507966 5040 scope.go:122] "RemoveContainer" containerID="3d8712cbeaf692c9d1c185a9ef1d1d0a5e954b762e1ba9f9a6ee923949a19020" Nov 01 07:00:16.135159 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:16.135074 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-045f69-29891940-jtpxg" event={"ID":"8daa9ba3-0fb1-4614-9013-8c907d18b85c","Type":"ContainerStarted","Data":"44ddadb6bcc43155ce15420e5c8118ea1db2f5408914d799aa955c154a17a162"} Nov 01 07:00:16.252768 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:16.252706 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-045f69-29891940-jtpxg" podStartSLOduration=16.252691329 podStartE2EDuration="16.252691329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:00:16.251849119 +0000 UTC m=+4494.906966286" watchObservedRunningTime="2026-11-01 07:00:16.252691329 +0000 UTC m=+4494.907808495" Nov 01 07:00:17.142252 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:17.142217 5040 generic.go:356] "Generic (PLEG): container finished" podID="8daa9ba3-0fb1-4614-9013-8c907d18b85c" containerID="44ddadb6bcc43155ce15420e5c8118ea1db2f5408914d799aa955c154a17a162" exitCode=60 Nov 01 07:00:17.142719 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:17.142288 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-045f69-29891940-jtpxg" event={"ID":"8daa9ba3-0fb1-4614-9013-8c907d18b85c","Type":"ContainerDied","Data":"44ddadb6bcc43155ce15420e5c8118ea1db2f5408914d799aa955c154a17a162"} Nov 01 07:00:17.142719 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:17.142347 5040 scope.go:122] "RemoveContainer" containerID="3d8712cbeaf692c9d1c185a9ef1d1d0a5e954b762e1ba9f9a6ee923949a19020" Nov 01 07:00:17.142719 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:17.142679 5040 scope.go:122] "RemoveContainer" containerID="44ddadb6bcc43155ce15420e5c8118ea1db2f5408914d799aa955c154a17a162" Nov 01 07:00:17.142958 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:00:17.142939 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-main-045f69-29891940-jtpxg_odh-ai-gateway-infra(8daa9ba3-0fb1-4614-9013-8c907d18b85c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-045f69-29891940-jtpxg" podUID="8daa9ba3-0fb1-4614-9013-8c907d18b85c" Nov 01 07:00:17.267884 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:17.267845 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-045f69-29891940-jtpxg"] Nov 01 07:00:18.283863 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:18.283834 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-045f69-29891940-jtpxg" Nov 01 07:00:18.394443 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:18.394406 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8daa9ba3-0fb1-4614-9013-8c907d18b85c-kube-api-access-tp45k\" (UniqueName: \"kubernetes.io/projected/8daa9ba3-0fb1-4614-9013-8c907d18b85c-kube-api-access-tp45k\") pod \"8daa9ba3-0fb1-4614-9013-8c907d18b85c\" (UID: \"8daa9ba3-0fb1-4614-9013-8c907d18b85c\") " Nov 01 07:00:18.394687 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:18.394458 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8daa9ba3-0fb1-4614-9013-8c907d18b85c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8daa9ba3-0fb1-4614-9013-8c907d18b85c-maas-api-service-ca\") pod \"8daa9ba3-0fb1-4614-9013-8c907d18b85c\" (UID: \"8daa9ba3-0fb1-4614-9013-8c907d18b85c\") " Nov 01 07:00:18.394869 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:18.394842 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8daa9ba3-0fb1-4614-9013-8c907d18b85c-maas-api-service-ca" pod "8daa9ba3-0fb1-4614-9013-8c907d18b85c" (UID: "8daa9ba3-0fb1-4614-9013-8c907d18b85c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:00:18.396918 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:18.396889 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8daa9ba3-0fb1-4614-9013-8c907d18b85c-kube-api-access-tp45k" pod "8daa9ba3-0fb1-4614-9013-8c907d18b85c" (UID: "8daa9ba3-0fb1-4614-9013-8c907d18b85c"). InnerVolumeSpecName "kube-api-access-tp45k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:00:18.495210 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:18.495173 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp45k\" (UniqueName: \"kubernetes.io/projected/8daa9ba3-0fb1-4614-9013-8c907d18b85c-kube-api-access-tp45k\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:00:18.495210 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:18.495206 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8daa9ba3-0fb1-4614-9013-8c907d18b85c-maas-api-service-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:00:19.156148 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:19.156114 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-045f69-29891940-jtpxg" Nov 01 07:00:19.156148 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:19.156125 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-045f69-29891940-jtpxg" event={"ID":"8daa9ba3-0fb1-4614-9013-8c907d18b85c","Type":"ContainerDied","Data":"d19c47b672fca620f464f4bdd82164f37ddfffe6cbad18630f1ba0de78a36b1e"} Nov 01 07:00:19.156415 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:19.156199 5040 scope.go:122] "RemoveContainer" containerID="44ddadb6bcc43155ce15420e5c8118ea1db2f5408914d799aa955c154a17a162" Nov 01 07:00:19.323557 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:19.323519 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-045f69-29891940-jtpxg"] Nov 01 07:00:19.329466 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:19.329438 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-045f69-29891940-jtpxg"] Nov 01 07:00:19.513995 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:19.513951 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8daa9ba3-0fb1-4614-9013-8c907d18b85c" path="/var/lib/kubelet/pods/8daa9ba3-0fb1-4614-9013-8c907d18b85c/volumes" Nov 01 07:00:22.326149 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:22.326118 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 07:00:22.339590 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:22.339567 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 07:00:24.191361 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.191323 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq"] Nov 01 07:00:24.191796 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.191720 5040 cpu_manager.go:414] "containerMap: removing container" podUID="8daa9ba3-0fb1-4614-9013-8c907d18b85c" containerName="cleanup" Nov 01 07:00:24.191796 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.191731 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8daa9ba3-0fb1-4614-9013-8c907d18b85c" containerName="cleanup" Nov 01 07:00:24.191796 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.191744 5040 cpu_manager.go:414] "containerMap: removing container" podUID="8daa9ba3-0fb1-4614-9013-8c907d18b85c" containerName="cleanup" Nov 01 07:00:24.191796 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.191749 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8daa9ba3-0fb1-4614-9013-8c907d18b85c" containerName="cleanup" Nov 01 07:00:24.191796 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.191769 5040 cpu_manager.go:414] "containerMap: removing container" podUID="8daa9ba3-0fb1-4614-9013-8c907d18b85c" containerName="cleanup" Nov 01 07:00:24.191796 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.191774 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8daa9ba3-0fb1-4614-9013-8c907d18b85c" containerName="cleanup" Nov 01 07:00:24.191999 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.191852 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="8daa9ba3-0fb1-4614-9013-8c907d18b85c" containerName="cleanup" Nov 01 07:00:24.191999 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.191860 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="8daa9ba3-0fb1-4614-9013-8c907d18b85c" containerName="cleanup" Nov 01 07:00:24.194932 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.194913 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" Nov 01 07:00:24.354408 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.354375 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnrtq\" (UID: \"09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" Nov 01 07:00:24.354643 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.354417 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnrtq\" (UID: \"09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" Nov 01 07:00:24.354643 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.354437 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnrtq\" (UID: \"09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" Nov 01 07:00:24.354643 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.354548 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnrtq\" (UID: \"09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" Nov 01 07:00:24.354643 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.354630 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnrtq\" (UID: \"09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" Nov 01 07:00:24.354811 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.354682 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnrtq\" (UID: \"09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" Nov 01 07:00:24.354811 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.354698 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnrtq\" (UID: \"09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" Nov 01 07:00:24.354811 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.354779 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnrtq\" (UID: \"09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" Nov 01 07:00:24.354923 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.354811 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-965qp\" (UniqueName: \"kubernetes.io/projected/09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7-kube-api-access-965qp\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnrtq\" (UID: \"09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" Nov 01 07:00:24.398690 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.398655 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq"] Nov 01 07:00:24.455528 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.455423 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnrtq\" (UID: \"09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" Nov 01 07:00:24.455528 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.455466 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnrtq\" (UID: \"09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" Nov 01 07:00:24.455755 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.455536 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnrtq\" (UID: \"09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" Nov 01 07:00:24.455755 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.455566 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-965qp\" (UniqueName: \"kubernetes.io/projected/09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7-kube-api-access-965qp\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnrtq\" (UID: \"09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" Nov 01 07:00:24.455755 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.455688 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnrtq\" (UID: \"09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" Nov 01 07:00:24.455755 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.455731 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnrtq\" (UID: \"09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" Nov 01 07:00:24.455755 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.455754 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnrtq\" (UID: \"09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" Nov 01 07:00:24.456029 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.455821 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnrtq\" (UID: \"09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" Nov 01 07:00:24.456029 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.455885 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnrtq\" (UID: \"09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" Nov 01 07:00:24.456029 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.456010 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnrtq\" (UID: \"09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" Nov 01 07:00:24.456207 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.456065 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnrtq\" (UID: \"09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" Nov 01 07:00:24.456271 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.456249 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnrtq\" (UID: \"09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" Nov 01 07:00:24.456329 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.456293 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnrtq\" (UID: \"09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" Nov 01 07:00:24.456412 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.456394 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnrtq\" (UID: \"09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" Nov 01 07:00:24.457799 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.457778 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnrtq\" (UID: \"09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" Nov 01 07:00:24.458150 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.458132 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnrtq\" (UID: \"09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" Nov 01 07:00:24.517595 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.517563 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-965qp\" (UniqueName: \"kubernetes.io/projected/09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7-kube-api-access-965qp\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnrtq\" (UID: \"09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" Nov 01 07:00:24.517754 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.517565 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnrtq\" (UID: \"09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" Nov 01 07:00:24.808021 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:24.807977 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" Nov 01 07:00:25.237519 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:25.237492 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq"] Nov 01 07:00:25.241965 ip-10-0-143-195 kubenswrapper[5040]: W1101 07:00:25.241930 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09bdb0f6_3f64_4e4b_8d03_f23a3cc0d5d7.slice/crio-1193ba98e8e7ab00b4a8e229f537084935d53ff5a6bdc5c208e0dea082273128 WatchSource:0}: Error finding container 1193ba98e8e7ab00b4a8e229f537084935d53ff5a6bdc5c208e0dea082273128: Status 404 returned error can't find the container with id 1193ba98e8e7ab00b4a8e229f537084935d53ff5a6bdc5c208e0dea082273128 Nov 01 07:00:25.243809 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:25.243775 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 07:00:25.243881 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:25.243867 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 07:00:25.243925 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:25.243899 5040 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 07:00:26.220834 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:26.220791 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" event={"ID":"09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7","Type":"ContainerStarted","Data":"1e6434acb964c5b4a71d57ce708d97c002b678d6ef072dd9c951e4ba9ef0571c"} Nov 01 07:00:26.221069 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:26.220857 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" event={"ID":"09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7","Type":"ContainerStarted","Data":"1193ba98e8e7ab00b4a8e229f537084935d53ff5a6bdc5c208e0dea082273128"} Nov 01 07:00:26.418503 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:26.405110 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" podStartSLOduration=2.4050879370000002 podStartE2EDuration="2.405087937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:00:26.392331837 +0000 UTC m=+4505.047449004" watchObservedRunningTime="2026-11-01 07:00:26.405087937 +0000 UTC m=+4505.060205107" Nov 01 07:00:26.814504 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:26.814379 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" Nov 01 07:00:26.818502 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:26.815574 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tnrtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Nov 01 07:00:26.818502 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:26.815635 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" podUID="09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Nov 01 07:00:27.814503 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:27.813928 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tnrtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Nov 01 07:00:27.814503 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:27.813995 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" podUID="09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Nov 01 07:00:28.816504 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:28.813921 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tnrtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Nov 01 07:00:28.816504 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:28.813987 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" podUID="09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Nov 01 07:00:29.818505 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:29.813905 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tnrtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Nov 01 07:00:29.818505 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:29.813985 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" podUID="09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Nov 01 07:00:30.812941 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:30.812899 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tnrtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Nov 01 07:00:30.813435 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:30.813393 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" podUID="09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Nov 01 07:00:31.808765 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:31.808725 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tnrtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Nov 01 07:00:31.809341 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:31.808804 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" podUID="09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Nov 01 07:00:32.808797 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:32.808758 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tnrtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Nov 01 07:00:32.809203 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:32.808824 5040 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" podUID="09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Nov 01 07:00:33.812863 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:33.812826 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" Nov 01 07:00:33.813317 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:33.813304 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" Nov 01 07:00:33.814261 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:00:33.814236 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" Nov 01 07:00:39.972621 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:00:39.972578 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:01:09.973904 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:01:09.973865 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:01:39.974182 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:01:39.974139 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:02:09.972378 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:02:09.972340 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:02:39.972635 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:02:39.972555 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:03:09.973699 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:03:09.973619 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:03:22.209283 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:22.209247 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9"] Nov 01 07:03:22.259446 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:22.259412 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4"] Nov 01 07:03:37.229305 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.229266 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9" podUID="351df948-ac15-469b-a606-c84b4b9cf3c9" containerName="main" containerID="cri-o://f9bd70072f90866697ff14de71ec75b51a62b0cd955921c0d3fd29510da59d46" gracePeriod=45 Nov 01 07:03:37.278723 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.278681 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4" podUID="d1211d6c-1cd3-4b33-a2a1-28125b936f91" containerName="main" containerID="cri-o://86457b1f501d41f7f950f01f010f3732bf7166f09c107d3e2e0bc7ce80e3343e" gracePeriod=45 Nov 01 07:03:37.468265 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.468227 5040 generic.go:356] "Generic (PLEG): container finished" podID="d1211d6c-1cd3-4b33-a2a1-28125b936f91" containerID="86457b1f501d41f7f950f01f010f3732bf7166f09c107d3e2e0bc7ce80e3343e" exitCode=0 Nov 01 07:03:37.468440 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.468293 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4" event={"ID":"d1211d6c-1cd3-4b33-a2a1-28125b936f91","Type":"ContainerDied","Data":"86457b1f501d41f7f950f01f010f3732bf7166f09c107d3e2e0bc7ce80e3343e"} Nov 01 07:03:37.470295 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.470274 5040 generic.go:356] "Generic (PLEG): container finished" podID="351df948-ac15-469b-a606-c84b4b9cf3c9" containerID="f9bd70072f90866697ff14de71ec75b51a62b0cd955921c0d3fd29510da59d46" exitCode=0 Nov 01 07:03:37.470405 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.470325 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9" event={"ID":"351df948-ac15-469b-a606-c84b4b9cf3c9","Type":"ContainerDied","Data":"f9bd70072f90866697ff14de71ec75b51a62b0cd955921c0d3fd29510da59d46"} Nov 01 07:03:37.534725 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.534701 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9" Nov 01 07:03:37.563366 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.563342 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4" Nov 01 07:03:37.664615 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.664570 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1211d6c-1cd3-4b33-a2a1-28125b936f91-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1211d6c-1cd3-4b33-a2a1-28125b936f91-model-cache\") pod \"d1211d6c-1cd3-4b33-a2a1-28125b936f91\" (UID: \"d1211d6c-1cd3-4b33-a2a1-28125b936f91\") " Nov 01 07:03:37.664870 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.664842 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1211d6c-1cd3-4b33-a2a1-28125b936f91-kube-api-access-w4hln\" (UniqueName: \"kubernetes.io/projected/d1211d6c-1cd3-4b33-a2a1-28125b936f91-kube-api-access-w4hln\") pod \"d1211d6c-1cd3-4b33-a2a1-28125b936f91\" (UID: \"d1211d6c-1cd3-4b33-a2a1-28125b936f91\") " Nov 01 07:03:37.664955 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.664896 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/351df948-ac15-469b-a606-c84b4b9cf3c9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/351df948-ac15-469b-a606-c84b4b9cf3c9-dshm\") pod \"351df948-ac15-469b-a606-c84b4b9cf3c9\" (UID: \"351df948-ac15-469b-a606-c84b4b9cf3c9\") " Nov 01 07:03:37.664955 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.664926 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1211d6c-1cd3-4b33-a2a1-28125b936f91-home\" (UniqueName: \"kubernetes.io/empty-dir/d1211d6c-1cd3-4b33-a2a1-28125b936f91-home\") pod \"d1211d6c-1cd3-4b33-a2a1-28125b936f91\" (UID: \"d1211d6c-1cd3-4b33-a2a1-28125b936f91\") " Nov 01 07:03:37.665083 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.665006 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1211d6c-1cd3-4b33-a2a1-28125b936f91-model-cache" pod "d1211d6c-1cd3-4b33-a2a1-28125b936f91" (UID: "d1211d6c-1cd3-4b33-a2a1-28125b936f91"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:03:37.665222 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.665203 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/351df948-ac15-469b-a606-c84b4b9cf3c9-home\" (UniqueName: \"kubernetes.io/empty-dir/351df948-ac15-469b-a606-c84b4b9cf3c9-home\") pod \"351df948-ac15-469b-a606-c84b4b9cf3c9\" (UID: \"351df948-ac15-469b-a606-c84b4b9cf3c9\") " Nov 01 07:03:37.665302 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.665256 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1211d6c-1cd3-4b33-a2a1-28125b936f91-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1211d6c-1cd3-4b33-a2a1-28125b936f91-dshm\") pod \"d1211d6c-1cd3-4b33-a2a1-28125b936f91\" (UID: \"d1211d6c-1cd3-4b33-a2a1-28125b936f91\") " Nov 01 07:03:37.665358 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.665311 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/351df948-ac15-469b-a606-c84b4b9cf3c9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/351df948-ac15-469b-a606-c84b4b9cf3c9-tmp-dir\") pod \"351df948-ac15-469b-a606-c84b4b9cf3c9\" (UID: \"351df948-ac15-469b-a606-c84b4b9cf3c9\") " Nov 01 07:03:37.665358 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.665344 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/351df948-ac15-469b-a606-c84b4b9cf3c9-kube-api-access-lllbv\" (UniqueName: \"kubernetes.io/projected/351df948-ac15-469b-a606-c84b4b9cf3c9-kube-api-access-lllbv\") pod \"351df948-ac15-469b-a606-c84b4b9cf3c9\" (UID: \"351df948-ac15-469b-a606-c84b4b9cf3c9\") " Nov 01 07:03:37.665466 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.665381 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1211d6c-1cd3-4b33-a2a1-28125b936f91-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1211d6c-1cd3-4b33-a2a1-28125b936f91-tmp-dir\") pod \"d1211d6c-1cd3-4b33-a2a1-28125b936f91\" (UID: \"d1211d6c-1cd3-4b33-a2a1-28125b936f91\") " Nov 01 07:03:37.665556 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.665520 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1211d6c-1cd3-4b33-a2a1-28125b936f91-tls-certs\" (UniqueName: \"kubernetes.io/secret/d1211d6c-1cd3-4b33-a2a1-28125b936f91-tls-certs\") pod \"d1211d6c-1cd3-4b33-a2a1-28125b936f91\" (UID: \"d1211d6c-1cd3-4b33-a2a1-28125b936f91\") " Nov 01 07:03:37.665556 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.665536 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1211d6c-1cd3-4b33-a2a1-28125b936f91-home" pod "d1211d6c-1cd3-4b33-a2a1-28125b936f91" (UID: "d1211d6c-1cd3-4b33-a2a1-28125b936f91"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:03:37.665665 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.665555 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351df948-ac15-469b-a606-c84b4b9cf3c9-home" pod "351df948-ac15-469b-a606-c84b4b9cf3c9" (UID: "351df948-ac15-469b-a606-c84b4b9cf3c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:03:37.665665 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.665582 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/351df948-ac15-469b-a606-c84b4b9cf3c9-tls-certs\" (UniqueName: \"kubernetes.io/secret/351df948-ac15-469b-a606-c84b4b9cf3c9-tls-certs\") pod \"351df948-ac15-469b-a606-c84b4b9cf3c9\" (UID: \"351df948-ac15-469b-a606-c84b4b9cf3c9\") " Nov 01 07:03:37.665780 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.665692 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1211d6c-1cd3-4b33-a2a1-28125b936f91-tmp-dir" pod "d1211d6c-1cd3-4b33-a2a1-28125b936f91" (UID: "d1211d6c-1cd3-4b33-a2a1-28125b936f91"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:03:37.665780 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.665692 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/351df948-ac15-469b-a606-c84b4b9cf3c9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/351df948-ac15-469b-a606-c84b4b9cf3c9-model-cache\") pod \"351df948-ac15-469b-a606-c84b4b9cf3c9\" (UID: \"351df948-ac15-469b-a606-c84b4b9cf3c9\") " Nov 01 07:03:37.665963 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.665935 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351df948-ac15-469b-a606-c84b4b9cf3c9-model-cache" pod "351df948-ac15-469b-a606-c84b4b9cf3c9" (UID: "351df948-ac15-469b-a606-c84b4b9cf3c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:03:37.666168 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.666149 5040 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1211d6c-1cd3-4b33-a2a1-28125b936f91-tmp-dir\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:03:37.666276 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.666173 5040 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/351df948-ac15-469b-a606-c84b4b9cf3c9-model-cache\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:03:37.666276 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.666187 5040 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1211d6c-1cd3-4b33-a2a1-28125b936f91-model-cache\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:03:37.666276 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.666202 5040 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1211d6c-1cd3-4b33-a2a1-28125b936f91-home\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:03:37.666276 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.666214 5040 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/351df948-ac15-469b-a606-c84b4b9cf3c9-home\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:03:37.666758 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.666732 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351df948-ac15-469b-a606-c84b4b9cf3c9-tmp-dir" pod "351df948-ac15-469b-a606-c84b4b9cf3c9" (UID: "351df948-ac15-469b-a606-c84b4b9cf3c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:03:37.668246 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.668222 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1211d6c-1cd3-4b33-a2a1-28125b936f91-kube-api-access-w4hln" pod "d1211d6c-1cd3-4b33-a2a1-28125b936f91" (UID: "d1211d6c-1cd3-4b33-a2a1-28125b936f91"). InnerVolumeSpecName "kube-api-access-w4hln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:03:37.668956 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.668934 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351df948-ac15-469b-a606-c84b4b9cf3c9-dshm" pod "351df948-ac15-469b-a606-c84b4b9cf3c9" (UID: "351df948-ac15-469b-a606-c84b4b9cf3c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:03:37.669108 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.669035 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1211d6c-1cd3-4b33-a2a1-28125b936f91-dshm" pod "d1211d6c-1cd3-4b33-a2a1-28125b936f91" (UID: "d1211d6c-1cd3-4b33-a2a1-28125b936f91"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:03:37.670248 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.670224 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351df948-ac15-469b-a606-c84b4b9cf3c9-kube-api-access-lllbv" pod "351df948-ac15-469b-a606-c84b4b9cf3c9" (UID: "351df948-ac15-469b-a606-c84b4b9cf3c9"). InnerVolumeSpecName "kube-api-access-lllbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:03:37.670710 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.670684 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1211d6c-1cd3-4b33-a2a1-28125b936f91-tls-certs" pod "d1211d6c-1cd3-4b33-a2a1-28125b936f91" (UID: "d1211d6c-1cd3-4b33-a2a1-28125b936f91"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:03:37.675565 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.675546 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/351df948-ac15-469b-a606-c84b4b9cf3c9-tls-certs" pod "351df948-ac15-469b-a606-c84b4b9cf3c9" (UID: "351df948-ac15-469b-a606-c84b4b9cf3c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:03:37.767077 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.767039 5040 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1211d6c-1cd3-4b33-a2a1-28125b936f91-dshm\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:03:37.767077 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.767070 5040 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/351df948-ac15-469b-a606-c84b4b9cf3c9-tmp-dir\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:03:37.767077 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.767080 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lllbv\" (UniqueName: \"kubernetes.io/projected/351df948-ac15-469b-a606-c84b4b9cf3c9-kube-api-access-lllbv\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:03:37.767319 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.767092 5040 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1211d6c-1cd3-4b33-a2a1-28125b936f91-tls-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:03:37.767319 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.767101 5040 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/351df948-ac15-469b-a606-c84b4b9cf3c9-tls-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:03:37.767319 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.767114 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4hln\" (UniqueName: \"kubernetes.io/projected/d1211d6c-1cd3-4b33-a2a1-28125b936f91-kube-api-access-w4hln\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:03:37.767319 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:37.767124 5040 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/351df948-ac15-469b-a606-c84b4b9cf3c9-dshm\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:03:38.477081 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:38.477046 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4" Nov 01 07:03:38.477642 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:38.477046 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4" event={"ID":"d1211d6c-1cd3-4b33-a2a1-28125b936f91","Type":"ContainerDied","Data":"c668c3a7b7a146436520a5a704567205d4d778f9f5dd181a2334097187eed0ca"} Nov 01 07:03:38.477642 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:38.477191 5040 scope.go:122] "RemoveContainer" containerID="86457b1f501d41f7f950f01f010f3732bf7166f09c107d3e2e0bc7ce80e3343e" Nov 01 07:03:38.478555 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:38.478524 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9" event={"ID":"351df948-ac15-469b-a606-c84b4b9cf3c9","Type":"ContainerDied","Data":"aaf81d54cc8753f44fe84d350b8acbd1d49812a899119e85a0ec7b39b290fbdf"} Nov 01 07:03:38.478645 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:38.478561 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9" Nov 01 07:03:38.487172 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:38.486994 5040 scope.go:122] "RemoveContainer" containerID="f9bd70072f90866697ff14de71ec75b51a62b0cd955921c0d3fd29510da59d46" Nov 01 07:03:38.643666 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:38.643633 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9"] Nov 01 07:03:38.693635 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:38.693604 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-main-045f69/premium-main-045f69-kserve-588987bfc4-hfpz9"] Nov 01 07:03:38.760077 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:38.760046 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4"] Nov 01 07:03:38.763175 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:38.763145 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-main-045f69/simulator-main-045f69-kserve-75c7b5f8cf-wn9h4"] Nov 01 07:03:39.515970 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:39.515936 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="351df948-ac15-469b-a606-c84b4b9cf3c9" path="/var/lib/kubelet/pods/351df948-ac15-469b-a606-c84b4b9cf3c9/volumes" Nov 01 07:03:39.516420 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:03:39.516403 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1211d6c-1cd3-4b33-a2a1-28125b936f91" path="/var/lib/kubelet/pods/d1211d6c-1cd3-4b33-a2a1-28125b936f91/volumes" Nov 01 07:03:39.971794 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:03:39.971716 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:04:09.973302 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:04:09.973263 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:04:39.973664 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:04:39.973626 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:05:09.972589 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:05:09.972546 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:05:17.733030 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:17.732996 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9af193c3-dr8kt"] Nov 01 07:05:17.733422 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:17.733382 5040 cpu_manager.go:414] "containerMap: removing container" podUID="d1211d6c-1cd3-4b33-a2a1-28125b936f91" containerName="main" Nov 01 07:05:17.733422 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:17.733394 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1211d6c-1cd3-4b33-a2a1-28125b936f91" containerName="main" Nov 01 07:05:17.733422 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:17.733410 5040 cpu_manager.go:414] "containerMap: removing container" podUID="351df948-ac15-469b-a606-c84b4b9cf3c9" containerName="main" Nov 01 07:05:17.733422 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:17.733417 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="351df948-ac15-469b-a606-c84b4b9cf3c9" containerName="main" Nov 01 07:05:17.733587 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:17.733519 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1211d6c-1cd3-4b33-a2a1-28125b936f91" containerName="main" Nov 01 07:05:17.733587 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:17.733530 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="351df948-ac15-469b-a606-c84b4b9cf3c9" containerName="main" Nov 01 07:05:17.733587 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:17.733537 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="8daa9ba3-0fb1-4614-9013-8c907d18b85c" containerName="cleanup" Nov 01 07:05:17.736599 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:17.736582 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9af193c3-dr8kt" Nov 01 07:05:17.842013 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:17.841978 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 07:05:17.842013 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:17.841989 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 07:05:17.842013 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:17.841990 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 07:05:17.842273 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:17.842087 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cf2w\"" Nov 01 07:05:17.844499 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:17.844468 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9af193c3-dr8kt"] Nov 01 07:05:17.876642 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:17.876606 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d271f46-c321-4712-9c69-5dbd9579ad51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9af193c3-dr8kt\" (UID: \"9d271f46-c321-4712-9c69-5dbd9579ad51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9af193c3-dr8kt" Nov 01 07:05:17.876813 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:17.876667 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x7hx\" (UniqueName: \"kubernetes.io/projected/9d271f46-c321-4712-9c69-5dbd9579ad51-kube-api-access-7x7hx\") pod \"maas-api-revoke-keys-e2e-ait-9af193c3-dr8kt\" (UID: \"9d271f46-c321-4712-9c69-5dbd9579ad51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9af193c3-dr8kt" Nov 01 07:05:17.876813 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:17.876795 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d271f46-c321-4712-9c69-5dbd9579ad51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9af193c3-dr8kt\" (UID: \"9d271f46-c321-4712-9c69-5dbd9579ad51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9af193c3-dr8kt" Nov 01 07:05:17.977582 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:17.977546 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x7hx\" (UniqueName: \"kubernetes.io/projected/9d271f46-c321-4712-9c69-5dbd9579ad51-kube-api-access-7x7hx\") pod \"maas-api-revoke-keys-e2e-ait-9af193c3-dr8kt\" (UID: \"9d271f46-c321-4712-9c69-5dbd9579ad51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9af193c3-dr8kt" Nov 01 07:05:17.977772 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:17.977593 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d271f46-c321-4712-9c69-5dbd9579ad51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9af193c3-dr8kt\" (UID: \"9d271f46-c321-4712-9c69-5dbd9579ad51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9af193c3-dr8kt" Nov 01 07:05:17.977772 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:17.977645 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d271f46-c321-4712-9c69-5dbd9579ad51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9af193c3-dr8kt\" (UID: \"9d271f46-c321-4712-9c69-5dbd9579ad51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9af193c3-dr8kt" Nov 01 07:05:17.978351 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:17.978317 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d271f46-c321-4712-9c69-5dbd9579ad51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9af193c3-dr8kt\" (UID: \"9d271f46-c321-4712-9c69-5dbd9579ad51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9af193c3-dr8kt" Nov 01 07:05:17.979990 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:17.979970 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d271f46-c321-4712-9c69-5dbd9579ad51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9af193c3-dr8kt\" (UID: \"9d271f46-c321-4712-9c69-5dbd9579ad51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9af193c3-dr8kt" Nov 01 07:05:18.038853 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:18.038820 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x7hx\" (UniqueName: \"kubernetes.io/projected/9d271f46-c321-4712-9c69-5dbd9579ad51-kube-api-access-7x7hx\") pod \"maas-api-revoke-keys-e2e-ait-9af193c3-dr8kt\" (UID: \"9d271f46-c321-4712-9c69-5dbd9579ad51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9af193c3-dr8kt" Nov 01 07:05:18.049659 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:18.049627 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9af193c3-dr8kt" Nov 01 07:05:18.421889 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:18.421843 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9af193c3-dr8kt"] Nov 01 07:05:18.423676 ip-10-0-143-195 kubenswrapper[5040]: W1101 07:05:18.423649 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d271f46_c321_4712_9c69_5dbd9579ad51.slice/crio-876938140d48e8a3fa681716bc9ac25ca5d44bfa6a56ea4538625f860cea0106 WatchSource:0}: Error finding container 876938140d48e8a3fa681716bc9ac25ca5d44bfa6a56ea4538625f860cea0106: Status 404 returned error can't find the container with id 876938140d48e8a3fa681716bc9ac25ca5d44bfa6a56ea4538625f860cea0106 Nov 01 07:05:19.005733 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:19.005690 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9af193c3-dr8kt" event={"ID":"9d271f46-c321-4712-9c69-5dbd9579ad51","Type":"ContainerStarted","Data":"39506bb9078e139cc3c7f89a12c035c1a54cb061b8171769543096e8ba647dd3"} Nov 01 07:05:19.006135 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:19.005776 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9af193c3-dr8kt" event={"ID":"9d271f46-c321-4712-9c69-5dbd9579ad51","Type":"ContainerStarted","Data":"876938140d48e8a3fa681716bc9ac25ca5d44bfa6a56ea4538625f860cea0106"} Nov 01 07:05:19.123419 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:19.123360 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9af193c3-dr8kt" podStartSLOduration=2.1233467089999998 podStartE2EDuration="2.123346709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:05:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:05:19.121739374 +0000 UTC m=+4797.776856542" watchObservedRunningTime="2026-11-01 07:05:19.123346709 +0000 UTC m=+4797.778463910" Nov 01 07:05:20.014218 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:20.014177 5040 generic.go:356] "Generic (PLEG): container finished" podID="9d271f46-c321-4712-9c69-5dbd9579ad51" containerID="39506bb9078e139cc3c7f89a12c035c1a54cb061b8171769543096e8ba647dd3" exitCode=0 Nov 01 07:05:20.014644 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:20.014265 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9af193c3-dr8kt" event={"ID":"9d271f46-c321-4712-9c69-5dbd9579ad51","Type":"ContainerDied","Data":"39506bb9078e139cc3c7f89a12c035c1a54cb061b8171769543096e8ba647dd3"} Nov 01 07:05:21.156688 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:21.156662 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9af193c3-dr8kt" Nov 01 07:05:21.208768 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:21.208732 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d271f46-c321-4712-9c69-5dbd9579ad51-kube-api-access-7x7hx\" (UniqueName: \"kubernetes.io/projected/9d271f46-c321-4712-9c69-5dbd9579ad51-kube-api-access-7x7hx\") pod \"9d271f46-c321-4712-9c69-5dbd9579ad51\" (UID: \"9d271f46-c321-4712-9c69-5dbd9579ad51\") " Nov 01 07:05:21.208768 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:21.208773 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d271f46-c321-4712-9c69-5dbd9579ad51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d271f46-c321-4712-9c69-5dbd9579ad51-maas-api-openshift-ca\") pod \"9d271f46-c321-4712-9c69-5dbd9579ad51\" (UID: \"9d271f46-c321-4712-9c69-5dbd9579ad51\") " Nov 01 07:05:21.209255 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:21.209223 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d271f46-c321-4712-9c69-5dbd9579ad51-maas-api-openshift-ca" pod "9d271f46-c321-4712-9c69-5dbd9579ad51" (UID: "9d271f46-c321-4712-9c69-5dbd9579ad51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:05:21.211067 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:21.211038 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d271f46-c321-4712-9c69-5dbd9579ad51-kube-api-access-7x7hx" pod "9d271f46-c321-4712-9c69-5dbd9579ad51" (UID: "9d271f46-c321-4712-9c69-5dbd9579ad51"). InnerVolumeSpecName "kube-api-access-7x7hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:05:21.309327 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:21.309232 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d271f46-c321-4712-9c69-5dbd9579ad51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d271f46-c321-4712-9c69-5dbd9579ad51-maas-api-xks-ca\") pod \"9d271f46-c321-4712-9c69-5dbd9579ad51\" (UID: \"9d271f46-c321-4712-9c69-5dbd9579ad51\") " Nov 01 07:05:21.309637 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:21.309610 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x7hx\" (UniqueName: \"kubernetes.io/projected/9d271f46-c321-4712-9c69-5dbd9579ad51-kube-api-access-7x7hx\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:05:21.309637 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:21.309639 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d271f46-c321-4712-9c69-5dbd9579ad51-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:05:21.311604 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:21.311579 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d271f46-c321-4712-9c69-5dbd9579ad51-maas-api-xks-ca" pod "9d271f46-c321-4712-9c69-5dbd9579ad51" (UID: "9d271f46-c321-4712-9c69-5dbd9579ad51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:05:21.410681 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:21.410649 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d271f46-c321-4712-9c69-5dbd9579ad51-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:05:22.027428 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:22.027395 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9af193c3-dr8kt" Nov 01 07:05:22.027635 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:22.027430 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9af193c3-dr8kt" event={"ID":"9d271f46-c321-4712-9c69-5dbd9579ad51","Type":"ContainerDied","Data":"876938140d48e8a3fa681716bc9ac25ca5d44bfa6a56ea4538625f860cea0106"} Nov 01 07:05:22.027635 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:22.027499 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="876938140d48e8a3fa681716bc9ac25ca5d44bfa6a56ea4538625f860cea0106" Nov 01 07:05:22.362889 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:22.362738 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 07:05:22.384522 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:22.380847 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 07:05:36.890724 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:36.890644 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9af193c3-dr8kt"] Nov 01 07:05:36.940959 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:36.940920 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9af193c3-dr8kt"] Nov 01 07:05:37.513669 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:05:37.513632 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d271f46-c321-4712-9c69-5dbd9579ad51" path="/var/lib/kubelet/pods/9d271f46-c321-4712-9c69-5dbd9579ad51/volumes" Nov 01 07:05:39.972910 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:05:39.972871 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:06:09.977138 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:06:09.977086 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:06:39.972311 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:06:39.972266 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:07:09.972740 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:07:09.972692 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:07:19.859654 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:07:19.859603 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tnrtq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 07:07:19.860445 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:07:19.860411 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" podUID="09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 07:07:39.972605 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:07:39.972567 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:08:09.971862 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:08:09.971822 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:08:15.245342 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:15.245300 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5771ebbb-qwc29"] Nov 01 07:08:15.245878 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:15.245859 5040 cpu_manager.go:414] "containerMap: removing container" podUID="9d271f46-c321-4712-9c69-5dbd9579ad51" containerName="revoke-keys" Nov 01 07:08:15.245935 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:15.245881 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d271f46-c321-4712-9c69-5dbd9579ad51" containerName="revoke-keys" Nov 01 07:08:15.245994 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:15.245982 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d271f46-c321-4712-9c69-5dbd9579ad51" containerName="revoke-keys" Nov 01 07:08:15.249210 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:15.249193 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5771ebbb-qwc29" Nov 01 07:08:15.325093 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:15.325056 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjd6k\" (UniqueName: \"kubernetes.io/projected/3b39ae85-fa77-4034-bde6-ecadedab7242-kube-api-access-wjd6k\") pod \"maas-api-revoke-sub-5771ebbb-qwc29\" (UID: \"3b39ae85-fa77-4034-bde6-ecadedab7242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5771ebbb-qwc29" Nov 01 07:08:15.325295 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:15.325118 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b39ae85-fa77-4034-bde6-ecadedab7242-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5771ebbb-qwc29\" (UID: \"3b39ae85-fa77-4034-bde6-ecadedab7242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5771ebbb-qwc29" Nov 01 07:08:15.325295 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:15.325207 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b39ae85-fa77-4034-bde6-ecadedab7242-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5771ebbb-qwc29\" (UID: \"3b39ae85-fa77-4034-bde6-ecadedab7242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5771ebbb-qwc29" Nov 01 07:08:15.354143 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:15.354103 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 07:08:15.354327 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:15.354115 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 07:08:15.354327 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:15.354173 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 07:08:15.354327 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:15.354186 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cf2w\"" Nov 01 07:08:15.356490 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:15.356457 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5771ebbb-qwc29"] Nov 01 07:08:15.426210 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:15.426177 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjd6k\" (UniqueName: \"kubernetes.io/projected/3b39ae85-fa77-4034-bde6-ecadedab7242-kube-api-access-wjd6k\") pod \"maas-api-revoke-sub-5771ebbb-qwc29\" (UID: \"3b39ae85-fa77-4034-bde6-ecadedab7242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5771ebbb-qwc29" Nov 01 07:08:15.426411 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:15.426235 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b39ae85-fa77-4034-bde6-ecadedab7242-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5771ebbb-qwc29\" (UID: \"3b39ae85-fa77-4034-bde6-ecadedab7242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5771ebbb-qwc29" Nov 01 07:08:15.426411 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:15.426259 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b39ae85-fa77-4034-bde6-ecadedab7242-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5771ebbb-qwc29\" (UID: \"3b39ae85-fa77-4034-bde6-ecadedab7242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5771ebbb-qwc29" Nov 01 07:08:15.426957 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:15.426933 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b39ae85-fa77-4034-bde6-ecadedab7242-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5771ebbb-qwc29\" (UID: \"3b39ae85-fa77-4034-bde6-ecadedab7242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5771ebbb-qwc29" Nov 01 07:08:15.428629 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:15.428610 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b39ae85-fa77-4034-bde6-ecadedab7242-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5771ebbb-qwc29\" (UID: \"3b39ae85-fa77-4034-bde6-ecadedab7242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5771ebbb-qwc29" Nov 01 07:08:15.488517 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:15.488468 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjd6k\" (UniqueName: \"kubernetes.io/projected/3b39ae85-fa77-4034-bde6-ecadedab7242-kube-api-access-wjd6k\") pod \"maas-api-revoke-sub-5771ebbb-qwc29\" (UID: \"3b39ae85-fa77-4034-bde6-ecadedab7242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5771ebbb-qwc29" Nov 01 07:08:15.561789 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:15.561755 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5771ebbb-qwc29" Nov 01 07:08:15.935172 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:15.935143 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5771ebbb-qwc29"] Nov 01 07:08:15.936563 ip-10-0-143-195 kubenswrapper[5040]: W1101 07:08:15.936536 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b39ae85_fa77_4034_bde6_ecadedab7242.slice/crio-3deb744fe40036b17b25d69f3f8562bf9c7dff3879465a862249833f87c05cc7 WatchSource:0}: Error finding container 3deb744fe40036b17b25d69f3f8562bf9c7dff3879465a862249833f87c05cc7: Status 404 returned error can't find the container with id 3deb744fe40036b17b25d69f3f8562bf9c7dff3879465a862249833f87c05cc7 Nov 01 07:08:16.031262 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:16.031228 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5771ebbb-qwc29" event={"ID":"3b39ae85-fa77-4034-bde6-ecadedab7242","Type":"ContainerStarted","Data":"3deb744fe40036b17b25d69f3f8562bf9c7dff3879465a862249833f87c05cc7"} Nov 01 07:08:17.037995 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:17.037952 5040 generic.go:356] "Generic (PLEG): container finished" podID="3b39ae85-fa77-4034-bde6-ecadedab7242" containerID="af345e8cf86bb239bb4c911020a66799b5c5fae087b48cba54ce7c7d818e5301" exitCode=0 Nov 01 07:08:17.038364 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:17.038000 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5771ebbb-qwc29" event={"ID":"3b39ae85-fa77-4034-bde6-ecadedab7242","Type":"ContainerDied","Data":"af345e8cf86bb239bb4c911020a66799b5c5fae087b48cba54ce7c7d818e5301"} Nov 01 07:08:18.175191 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:18.175163 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5771ebbb-qwc29" Nov 01 07:08:18.248839 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:18.248807 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b39ae85-fa77-4034-bde6-ecadedab7242-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b39ae85-fa77-4034-bde6-ecadedab7242-maas-api-xks-ca\") pod \"3b39ae85-fa77-4034-bde6-ecadedab7242\" (UID: \"3b39ae85-fa77-4034-bde6-ecadedab7242\") " Nov 01 07:08:18.249027 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:18.248852 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b39ae85-fa77-4034-bde6-ecadedab7242-kube-api-access-wjd6k\" (UniqueName: \"kubernetes.io/projected/3b39ae85-fa77-4034-bde6-ecadedab7242-kube-api-access-wjd6k\") pod \"3b39ae85-fa77-4034-bde6-ecadedab7242\" (UID: \"3b39ae85-fa77-4034-bde6-ecadedab7242\") " Nov 01 07:08:18.249027 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:18.248905 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b39ae85-fa77-4034-bde6-ecadedab7242-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b39ae85-fa77-4034-bde6-ecadedab7242-maas-api-openshift-ca\") pod \"3b39ae85-fa77-4034-bde6-ecadedab7242\" (UID: \"3b39ae85-fa77-4034-bde6-ecadedab7242\") " Nov 01 07:08:18.249349 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:18.249318 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b39ae85-fa77-4034-bde6-ecadedab7242-maas-api-openshift-ca" pod "3b39ae85-fa77-4034-bde6-ecadedab7242" (UID: "3b39ae85-fa77-4034-bde6-ecadedab7242"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:08:18.251022 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:18.250997 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b39ae85-fa77-4034-bde6-ecadedab7242-maas-api-xks-ca" pod "3b39ae85-fa77-4034-bde6-ecadedab7242" (UID: "3b39ae85-fa77-4034-bde6-ecadedab7242"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:08:18.251022 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:18.251006 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b39ae85-fa77-4034-bde6-ecadedab7242-kube-api-access-wjd6k" pod "3b39ae85-fa77-4034-bde6-ecadedab7242" (UID: "3b39ae85-fa77-4034-bde6-ecadedab7242"). InnerVolumeSpecName "kube-api-access-wjd6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:08:18.349923 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:18.349847 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b39ae85-fa77-4034-bde6-ecadedab7242-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:18.349923 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:18.349877 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjd6k\" (UniqueName: \"kubernetes.io/projected/3b39ae85-fa77-4034-bde6-ecadedab7242-kube-api-access-wjd6k\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:18.349923 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:18.349888 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b39ae85-fa77-4034-bde6-ecadedab7242-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:19.049883 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:19.049850 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5771ebbb-qwc29" Nov 01 07:08:19.049883 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:19.049875 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5771ebbb-qwc29" event={"ID":"3b39ae85-fa77-4034-bde6-ecadedab7242","Type":"ContainerDied","Data":"3deb744fe40036b17b25d69f3f8562bf9c7dff3879465a862249833f87c05cc7"} Nov 01 07:08:19.050111 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:19.049917 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3deb744fe40036b17b25d69f3f8562bf9c7dff3879465a862249833f87c05cc7" Nov 01 07:08:34.875696 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:34.875605 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tnrtq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 07:08:34.875696 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:08:34.875685 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" podUID="09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 07:08:39.972896 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:08:39.972855 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:09:03.924553 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:03.924511 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88e79a77-vkxc8"] Nov 01 07:09:03.925042 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:03.924928 5040 cpu_manager.go:414] "containerMap: removing container" podUID="3b39ae85-fa77-4034-bde6-ecadedab7242" containerName="revoke-keys" Nov 01 07:09:03.925042 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:03.924939 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b39ae85-fa77-4034-bde6-ecadedab7242" containerName="revoke-keys" Nov 01 07:09:03.925042 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:03.924999 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b39ae85-fa77-4034-bde6-ecadedab7242" containerName="revoke-keys" Nov 01 07:09:03.927966 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:03.927949 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e79a77-vkxc8" Nov 01 07:09:03.982695 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:03.982649 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 07:09:04.003112 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:04.003075 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d8915e4-9ce5-41ee-92a3-9f6641a3c285-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88e79a77-vkxc8\" (UID: \"0d8915e4-9ce5-41ee-92a3-9f6641a3c285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e79a77-vkxc8" Nov 01 07:09:04.003112 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:04.003117 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc2m8\" (UniqueName: \"kubernetes.io/projected/0d8915e4-9ce5-41ee-92a3-9f6641a3c285-kube-api-access-qc2m8\") pod \"maas-api-revoke-sub-88e79a77-vkxc8\" (UID: \"0d8915e4-9ce5-41ee-92a3-9f6641a3c285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e79a77-vkxc8" Nov 01 07:09:04.003376 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:04.003247 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d8915e4-9ce5-41ee-92a3-9f6641a3c285-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88e79a77-vkxc8\" (UID: \"0d8915e4-9ce5-41ee-92a3-9f6641a3c285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e79a77-vkxc8" Nov 01 07:09:04.033716 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:04.033676 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 07:09:04.033716 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:04.033697 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cf2w\"" Nov 01 07:09:04.033716 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:04.033685 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 07:09:04.036013 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:04.035995 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88e79a77-vkxc8"] Nov 01 07:09:04.104558 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:04.104527 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc2m8\" (UniqueName: \"kubernetes.io/projected/0d8915e4-9ce5-41ee-92a3-9f6641a3c285-kube-api-access-qc2m8\") pod \"maas-api-revoke-sub-88e79a77-vkxc8\" (UID: \"0d8915e4-9ce5-41ee-92a3-9f6641a3c285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e79a77-vkxc8" Nov 01 07:09:04.104719 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:04.104610 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d8915e4-9ce5-41ee-92a3-9f6641a3c285-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88e79a77-vkxc8\" (UID: \"0d8915e4-9ce5-41ee-92a3-9f6641a3c285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e79a77-vkxc8" Nov 01 07:09:04.104767 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:04.104719 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d8915e4-9ce5-41ee-92a3-9f6641a3c285-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88e79a77-vkxc8\" (UID: \"0d8915e4-9ce5-41ee-92a3-9f6641a3c285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e79a77-vkxc8" Nov 01 07:09:04.105255 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:04.105232 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d8915e4-9ce5-41ee-92a3-9f6641a3c285-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88e79a77-vkxc8\" (UID: \"0d8915e4-9ce5-41ee-92a3-9f6641a3c285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e79a77-vkxc8" Nov 01 07:09:04.107078 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:04.107057 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d8915e4-9ce5-41ee-92a3-9f6641a3c285-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88e79a77-vkxc8\" (UID: \"0d8915e4-9ce5-41ee-92a3-9f6641a3c285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e79a77-vkxc8" Nov 01 07:09:04.166849 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:04.166815 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc2m8\" (UniqueName: \"kubernetes.io/projected/0d8915e4-9ce5-41ee-92a3-9f6641a3c285-kube-api-access-qc2m8\") pod \"maas-api-revoke-sub-88e79a77-vkxc8\" (UID: \"0d8915e4-9ce5-41ee-92a3-9f6641a3c285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e79a77-vkxc8" Nov 01 07:09:04.240723 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:04.240637 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e79a77-vkxc8" Nov 01 07:09:04.612924 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:04.612895 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88e79a77-vkxc8"] Nov 01 07:09:04.614386 ip-10-0-143-195 kubenswrapper[5040]: W1101 07:09:04.614358 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d8915e4_9ce5_41ee_92a3_9f6641a3c285.slice/crio-997f6afd786333ba7574982a5ace11b8eb33b756c7b2771d87bc7535d639f383 WatchSource:0}: Error finding container 997f6afd786333ba7574982a5ace11b8eb33b756c7b2771d87bc7535d639f383: Status 404 returned error can't find the container with id 997f6afd786333ba7574982a5ace11b8eb33b756c7b2771d87bc7535d639f383 Nov 01 07:09:05.339148 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:05.339107 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e79a77-vkxc8" event={"ID":"0d8915e4-9ce5-41ee-92a3-9f6641a3c285","Type":"ContainerStarted","Data":"adeea783356a0d850f3532b2b4d3b076bb9a7db15efa212ad7412b3201664066"} Nov 01 07:09:05.339588 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:05.339167 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e79a77-vkxc8" event={"ID":"0d8915e4-9ce5-41ee-92a3-9f6641a3c285","Type":"ContainerStarted","Data":"997f6afd786333ba7574982a5ace11b8eb33b756c7b2771d87bc7535d639f383"} Nov 01 07:09:05.460299 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:05.460247 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e79a77-vkxc8" podStartSLOduration=2.460232704 podStartE2EDuration="2.460232704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:09:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:09:05.458875355 +0000 UTC m=+5024.113992537" watchObservedRunningTime="2026-11-01 07:09:05.460232704 +0000 UTC m=+5024.115349871" Nov 01 07:09:06.345933 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:06.345897 5040 generic.go:356] "Generic (PLEG): container finished" podID="0d8915e4-9ce5-41ee-92a3-9f6641a3c285" containerID="adeea783356a0d850f3532b2b4d3b076bb9a7db15efa212ad7412b3201664066" exitCode=0 Nov 01 07:09:06.346352 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:06.345988 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e79a77-vkxc8" event={"ID":"0d8915e4-9ce5-41ee-92a3-9f6641a3c285","Type":"ContainerDied","Data":"adeea783356a0d850f3532b2b4d3b076bb9a7db15efa212ad7412b3201664066"} Nov 01 07:09:07.492209 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:07.492185 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e79a77-vkxc8" Nov 01 07:09:07.538511 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:07.538469 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d8915e4-9ce5-41ee-92a3-9f6641a3c285-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d8915e4-9ce5-41ee-92a3-9f6641a3c285-maas-api-openshift-ca\") pod \"0d8915e4-9ce5-41ee-92a3-9f6641a3c285\" (UID: \"0d8915e4-9ce5-41ee-92a3-9f6641a3c285\") " Nov 01 07:09:07.538661 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:07.538534 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d8915e4-9ce5-41ee-92a3-9f6641a3c285-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d8915e4-9ce5-41ee-92a3-9f6641a3c285-maas-api-xks-ca\") pod \"0d8915e4-9ce5-41ee-92a3-9f6641a3c285\" (UID: \"0d8915e4-9ce5-41ee-92a3-9f6641a3c285\") " Nov 01 07:09:07.538661 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:07.538622 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d8915e4-9ce5-41ee-92a3-9f6641a3c285-kube-api-access-qc2m8\" (UniqueName: \"kubernetes.io/projected/0d8915e4-9ce5-41ee-92a3-9f6641a3c285-kube-api-access-qc2m8\") pod \"0d8915e4-9ce5-41ee-92a3-9f6641a3c285\" (UID: \"0d8915e4-9ce5-41ee-92a3-9f6641a3c285\") " Nov 01 07:09:07.538831 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:07.538806 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d8915e4-9ce5-41ee-92a3-9f6641a3c285-maas-api-openshift-ca" pod "0d8915e4-9ce5-41ee-92a3-9f6641a3c285" (UID: "0d8915e4-9ce5-41ee-92a3-9f6641a3c285"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:09:07.540620 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:07.540595 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d8915e4-9ce5-41ee-92a3-9f6641a3c285-kube-api-access-qc2m8" pod "0d8915e4-9ce5-41ee-92a3-9f6641a3c285" (UID: "0d8915e4-9ce5-41ee-92a3-9f6641a3c285"). InnerVolumeSpecName "kube-api-access-qc2m8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:09:07.540715 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:07.540666 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d8915e4-9ce5-41ee-92a3-9f6641a3c285-maas-api-xks-ca" pod "0d8915e4-9ce5-41ee-92a3-9f6641a3c285" (UID: "0d8915e4-9ce5-41ee-92a3-9f6641a3c285"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:09:07.639377 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:07.639299 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d8915e4-9ce5-41ee-92a3-9f6641a3c285-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:07.639377 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:07.639328 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc2m8\" (UniqueName: \"kubernetes.io/projected/0d8915e4-9ce5-41ee-92a3-9f6641a3c285-kube-api-access-qc2m8\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:07.639377 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:07.639339 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d8915e4-9ce5-41ee-92a3-9f6641a3c285-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:08.360790 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:08.360756 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e79a77-vkxc8" Nov 01 07:09:08.360973 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:08.360786 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e79a77-vkxc8" event={"ID":"0d8915e4-9ce5-41ee-92a3-9f6641a3c285","Type":"ContainerDied","Data":"997f6afd786333ba7574982a5ace11b8eb33b756c7b2771d87bc7535d639f383"} Nov 01 07:09:08.360973 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:08.360837 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="997f6afd786333ba7574982a5ace11b8eb33b756c7b2771d87bc7535d639f383" Nov 01 07:09:09.973236 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:09:09.973195 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:09:39.972326 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:09:39.972275 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:09:49.866675 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:49.866621 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tnrtq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 07:09:49.867367 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:09:49.866698 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" podUID="09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 07:10:09.941729 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:10:09.941694 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd"] Nov 01 07:10:09.942188 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:10:09.942158 5040 cpu_manager.go:414] "containerMap: removing container" podUID="0d8915e4-9ce5-41ee-92a3-9f6641a3c285" containerName="revoke-keys" Nov 01 07:10:09.942188 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:10:09.942179 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d8915e4-9ce5-41ee-92a3-9f6641a3c285" containerName="revoke-keys" Nov 01 07:10:09.942346 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:10:09.942260 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d8915e4-9ce5-41ee-92a3-9f6641a3c285" containerName="revoke-keys" Nov 01 07:10:09.945186 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:10:09.945167 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd" Nov 01 07:10:09.975148 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:10:09.975109 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:10:10.038155 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:10:10.038122 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff14bf0d-49ba-43b1-a64c-61e3482983b8-home\") pod \"e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd\" (UID: \"ff14bf0d-49ba-43b1-a64c-61e3482983b8\") " pod="llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd" Nov 01 07:10:10.038331 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:10:10.038168 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff14bf0d-49ba-43b1-a64c-61e3482983b8-dshm\") pod \"e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd\" (UID: \"ff14bf0d-49ba-43b1-a64c-61e3482983b8\") " pod="llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd" Nov 01 07:10:10.038331 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:10:10.038198 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff14bf0d-49ba-43b1-a64c-61e3482983b8-tmp-dir\") pod \"e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd\" (UID: \"ff14bf0d-49ba-43b1-a64c-61e3482983b8\") " pod="llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd" Nov 01 07:10:10.038414 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:10:10.038344 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff14bf0d-49ba-43b1-a64c-61e3482983b8-model-cache\") pod \"e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd\" (UID: \"ff14bf0d-49ba-43b1-a64c-61e3482983b8\") " pod="llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd" Nov 01 07:10:10.038458 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:10:10.038418 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pfq2\" (UniqueName: \"kubernetes.io/projected/ff14bf0d-49ba-43b1-a64c-61e3482983b8-kube-api-access-9pfq2\") pod \"e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd\" (UID: \"ff14bf0d-49ba-43b1-a64c-61e3482983b8\") " pod="llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd" Nov 01 07:10:10.038519 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:10:10.038460 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff14bf0d-49ba-43b1-a64c-61e3482983b8-tls-certs\") pod \"e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd\" (UID: \"ff14bf0d-49ba-43b1-a64c-61e3482983b8\") " pod="llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd" Nov 01 07:10:10.051344 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:10:10.051314 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-8d2e6917-kserve-self-signed-certs\"" Nov 01 07:10:10.053757 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:10:10.053734 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd"] Nov 01 07:10:10.139201 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:10:10.139164 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff14bf0d-49ba-43b1-a64c-61e3482983b8-model-cache\") pod \"e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd\" (UID: \"ff14bf0d-49ba-43b1-a64c-61e3482983b8\") " pod="llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd" Nov 01 07:10:10.139395 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:10:10.139212 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pfq2\" (UniqueName: \"kubernetes.io/projected/ff14bf0d-49ba-43b1-a64c-61e3482983b8-kube-api-access-9pfq2\") pod \"e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd\" (UID: \"ff14bf0d-49ba-43b1-a64c-61e3482983b8\") " pod="llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd" Nov 01 07:10:10.139395 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:10:10.139258 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff14bf0d-49ba-43b1-a64c-61e3482983b8-tls-certs\") pod \"e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd\" (UID: \"ff14bf0d-49ba-43b1-a64c-61e3482983b8\") " pod="llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd" Nov 01 07:10:10.139395 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:10:10.139316 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff14bf0d-49ba-43b1-a64c-61e3482983b8-home\") pod \"e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd\" (UID: \"ff14bf0d-49ba-43b1-a64c-61e3482983b8\") " pod="llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd" Nov 01 07:10:10.139395 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:10:10.139357 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff14bf0d-49ba-43b1-a64c-61e3482983b8-dshm\") pod \"e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd\" (UID: \"ff14bf0d-49ba-43b1-a64c-61e3482983b8\") " pod="llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd" Nov 01 07:10:10.139395 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:10:10.139385 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff14bf0d-49ba-43b1-a64c-61e3482983b8-tmp-dir\") pod \"e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd\" (UID: \"ff14bf0d-49ba-43b1-a64c-61e3482983b8\") " pod="llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd" Nov 01 07:10:10.139748 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:10:10.139718 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff14bf0d-49ba-43b1-a64c-61e3482983b8-model-cache\") pod \"e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd\" (UID: \"ff14bf0d-49ba-43b1-a64c-61e3482983b8\") " pod="llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd" Nov 01 07:10:10.139807 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:10:10.139742 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff14bf0d-49ba-43b1-a64c-61e3482983b8-home\") pod \"e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd\" (UID: \"ff14bf0d-49ba-43b1-a64c-61e3482983b8\") " pod="llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd" Nov 01 07:10:10.139807 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:10:10.139774 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff14bf0d-49ba-43b1-a64c-61e3482983b8-tmp-dir\") pod \"e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd\" (UID: \"ff14bf0d-49ba-43b1-a64c-61e3482983b8\") " pod="llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd" Nov 01 07:10:10.141668 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:10:10.141647 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff14bf0d-49ba-43b1-a64c-61e3482983b8-dshm\") pod \"e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd\" (UID: \"ff14bf0d-49ba-43b1-a64c-61e3482983b8\") " pod="llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd" Nov 01 07:10:10.141946 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:10:10.141928 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff14bf0d-49ba-43b1-a64c-61e3482983b8-tls-certs\") pod \"e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd\" (UID: \"ff14bf0d-49ba-43b1-a64c-61e3482983b8\") " pod="llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd" Nov 01 07:10:10.204058 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:10:10.203976 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pfq2\" (UniqueName: \"kubernetes.io/projected/ff14bf0d-49ba-43b1-a64c-61e3482983b8-kube-api-access-9pfq2\") pod \"e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd\" (UID: \"ff14bf0d-49ba-43b1-a64c-61e3482983b8\") " pod="llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd" Nov 01 07:10:10.257029 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:10:10.256986 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd" Nov 01 07:10:10.638120 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:10:10.638093 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd"] Nov 01 07:10:10.639685 ip-10-0-143-195 kubenswrapper[5040]: W1101 07:10:10.639661 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff14bf0d_49ba_43b1_a64c_61e3482983b8.slice/crio-77b929928ab97183535a486be42827c7d715c356de4e9b7de388b5ed027ca70b WatchSource:0}: Error finding container 77b929928ab97183535a486be42827c7d715c356de4e9b7de388b5ed027ca70b: Status 404 returned error can't find the container with id 77b929928ab97183535a486be42827c7d715c356de4e9b7de388b5ed027ca70b Nov 01 07:10:10.696338 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:10:10.696296 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd" event={"ID":"ff14bf0d-49ba-43b1-a64c-61e3482983b8","Type":"ContainerStarted","Data":"77b929928ab97183535a486be42827c7d715c356de4e9b7de388b5ed027ca70b"} Nov 01 07:10:11.703698 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:10:11.703660 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd" event={"ID":"ff14bf0d-49ba-43b1-a64c-61e3482983b8","Type":"ContainerStarted","Data":"f4b20338e48a1aa03e79c896a424ca4c94cbfdb390261d0be3cf89c94be0a645"} Nov 01 07:10:11.824400 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:10:11.824334 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd" podStartSLOduration=2.824312192 podStartE2EDuration="2.824312192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:10:11.822712541 +0000 UTC m=+5090.477829763" watchObservedRunningTime="2026-11-01 07:10:11.824312192 +0000 UTC m=+5090.479429372" Nov 01 07:10:12.257590 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:10:12.257551 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd" Nov 01 07:10:20.258012 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:10:20.257969 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd" Nov 01 07:10:20.270313 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:10:20.270286 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd" Nov 01 07:10:20.819305 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:10:20.819270 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd" Nov 01 07:10:22.404552 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:10:22.404427 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 07:10:22.419901 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:10:22.419880 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 07:10:39.973397 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:10:39.973355 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:11:09.972284 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:11:09.972243 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:11:13.414699 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:11:13.414662 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd"] Nov 01 07:11:24.486776 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:11:24.486739 5040 scope.go:122] "RemoveContainer" containerID="39506bb9078e139cc3c7f89a12c035c1a54cb061b8171769543096e8ba647dd3" Nov 01 07:11:28.435195 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:11:28.435157 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd" podUID="ff14bf0d-49ba-43b1-a64c-61e3482983b8" containerName="main" containerID="cri-o://f4b20338e48a1aa03e79c896a424ca4c94cbfdb390261d0be3cf89c94be0a645" gracePeriod=45 Nov 01 07:11:28.690425 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:11:28.690359 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd" Nov 01 07:11:28.735629 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:11:28.735592 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff14bf0d-49ba-43b1-a64c-61e3482983b8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff14bf0d-49ba-43b1-a64c-61e3482983b8-model-cache\") pod \"ff14bf0d-49ba-43b1-a64c-61e3482983b8\" (UID: \"ff14bf0d-49ba-43b1-a64c-61e3482983b8\") " Nov 01 07:11:28.735629 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:11:28.735633 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff14bf0d-49ba-43b1-a64c-61e3482983b8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff14bf0d-49ba-43b1-a64c-61e3482983b8-tmp-dir\") pod \"ff14bf0d-49ba-43b1-a64c-61e3482983b8\" (UID: \"ff14bf0d-49ba-43b1-a64c-61e3482983b8\") " Nov 01 07:11:28.735801 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:11:28.735664 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff14bf0d-49ba-43b1-a64c-61e3482983b8-tls-certs\" (UniqueName: \"kubernetes.io/secret/ff14bf0d-49ba-43b1-a64c-61e3482983b8-tls-certs\") pod \"ff14bf0d-49ba-43b1-a64c-61e3482983b8\" (UID: \"ff14bf0d-49ba-43b1-a64c-61e3482983b8\") " Nov 01 07:11:28.735801 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:11:28.735702 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff14bf0d-49ba-43b1-a64c-61e3482983b8-kube-api-access-9pfq2\" (UniqueName: \"kubernetes.io/projected/ff14bf0d-49ba-43b1-a64c-61e3482983b8-kube-api-access-9pfq2\") pod \"ff14bf0d-49ba-43b1-a64c-61e3482983b8\" (UID: \"ff14bf0d-49ba-43b1-a64c-61e3482983b8\") " Nov 01 07:11:28.735801 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:11:28.735723 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff14bf0d-49ba-43b1-a64c-61e3482983b8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff14bf0d-49ba-43b1-a64c-61e3482983b8-dshm\") pod \"ff14bf0d-49ba-43b1-a64c-61e3482983b8\" (UID: \"ff14bf0d-49ba-43b1-a64c-61e3482983b8\") " Nov 01 07:11:28.735801 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:11:28.735770 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff14bf0d-49ba-43b1-a64c-61e3482983b8-home\" (UniqueName: \"kubernetes.io/empty-dir/ff14bf0d-49ba-43b1-a64c-61e3482983b8-home\") pod \"ff14bf0d-49ba-43b1-a64c-61e3482983b8\" (UID: \"ff14bf0d-49ba-43b1-a64c-61e3482983b8\") " Nov 01 07:11:28.736027 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:11:28.735880 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff14bf0d-49ba-43b1-a64c-61e3482983b8-model-cache" pod "ff14bf0d-49ba-43b1-a64c-61e3482983b8" (UID: "ff14bf0d-49ba-43b1-a64c-61e3482983b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:11:28.736027 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:11:28.735984 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff14bf0d-49ba-43b1-a64c-61e3482983b8-tmp-dir" pod "ff14bf0d-49ba-43b1-a64c-61e3482983b8" (UID: "ff14bf0d-49ba-43b1-a64c-61e3482983b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:11:28.736151 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:11:28.736081 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff14bf0d-49ba-43b1-a64c-61e3482983b8-home" pod "ff14bf0d-49ba-43b1-a64c-61e3482983b8" (UID: "ff14bf0d-49ba-43b1-a64c-61e3482983b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:11:28.736151 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:11:28.736147 5040 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff14bf0d-49ba-43b1-a64c-61e3482983b8-model-cache\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:11:28.736227 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:11:28.736164 5040 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff14bf0d-49ba-43b1-a64c-61e3482983b8-tmp-dir\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:11:28.738083 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:11:28.738052 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff14bf0d-49ba-43b1-a64c-61e3482983b8-dshm" pod "ff14bf0d-49ba-43b1-a64c-61e3482983b8" (UID: "ff14bf0d-49ba-43b1-a64c-61e3482983b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:11:28.738198 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:11:28.738105 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff14bf0d-49ba-43b1-a64c-61e3482983b8-tls-certs" pod "ff14bf0d-49ba-43b1-a64c-61e3482983b8" (UID: "ff14bf0d-49ba-43b1-a64c-61e3482983b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:11:28.738198 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:11:28.738114 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff14bf0d-49ba-43b1-a64c-61e3482983b8-kube-api-access-9pfq2" pod "ff14bf0d-49ba-43b1-a64c-61e3482983b8" (UID: "ff14bf0d-49ba-43b1-a64c-61e3482983b8"). InnerVolumeSpecName "kube-api-access-9pfq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:11:28.836864 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:11:28.836819 5040 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff14bf0d-49ba-43b1-a64c-61e3482983b8-home\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:11:28.836864 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:11:28.836859 5040 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff14bf0d-49ba-43b1-a64c-61e3482983b8-tls-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:11:28.836864 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:11:28.836872 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pfq2\" (UniqueName: \"kubernetes.io/projected/ff14bf0d-49ba-43b1-a64c-61e3482983b8-kube-api-access-9pfq2\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:11:28.837125 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:11:28.836885 5040 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff14bf0d-49ba-43b1-a64c-61e3482983b8-dshm\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:11:29.221822 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:11:29.221786 5040 generic.go:356] "Generic (PLEG): container finished" podID="ff14bf0d-49ba-43b1-a64c-61e3482983b8" containerID="f4b20338e48a1aa03e79c896a424ca4c94cbfdb390261d0be3cf89c94be0a645" exitCode=0 Nov 01 07:11:29.222045 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:11:29.221828 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd" Nov 01 07:11:29.222045 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:11:29.221869 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd" event={"ID":"ff14bf0d-49ba-43b1-a64c-61e3482983b8","Type":"ContainerDied","Data":"f4b20338e48a1aa03e79c896a424ca4c94cbfdb390261d0be3cf89c94be0a645"} Nov 01 07:11:29.222045 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:11:29.221918 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd" event={"ID":"ff14bf0d-49ba-43b1-a64c-61e3482983b8","Type":"ContainerDied","Data":"77b929928ab97183535a486be42827c7d715c356de4e9b7de388b5ed027ca70b"} Nov 01 07:11:29.222045 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:11:29.221943 5040 scope.go:122] "RemoveContainer" containerID="f4b20338e48a1aa03e79c896a424ca4c94cbfdb390261d0be3cf89c94be0a645" Nov 01 07:11:29.232242 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:11:29.232222 5040 scope.go:122] "RemoveContainer" containerID="f4b20338e48a1aa03e79c896a424ca4c94cbfdb390261d0be3cf89c94be0a645" Nov 01 07:11:29.232668 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:11:29.232645 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4b20338e48a1aa03e79c896a424ca4c94cbfdb390261d0be3cf89c94be0a645\": container with ID starting with f4b20338e48a1aa03e79c896a424ca4c94cbfdb390261d0be3cf89c94be0a645 not found: ID does not exist" containerID="f4b20338e48a1aa03e79c896a424ca4c94cbfdb390261d0be3cf89c94be0a645" Nov 01 07:11:29.232745 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:11:29.232677 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b20338e48a1aa03e79c896a424ca4c94cbfdb390261d0be3cf89c94be0a645"} err="failed to get container status \"f4b20338e48a1aa03e79c896a424ca4c94cbfdb390261d0be3cf89c94be0a645\": rpc error: code = NotFound desc = could not find container \"f4b20338e48a1aa03e79c896a424ca4c94cbfdb390261d0be3cf89c94be0a645\": container with ID starting with f4b20338e48a1aa03e79c896a424ca4c94cbfdb390261d0be3cf89c94be0a645 not found: ID does not exist" Nov 01 07:11:29.389113 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:11:29.389074 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd"] Nov 01 07:11:29.439517 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:11:29.439465 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-8d2e6917-kserve-644f47c5d5-c2xdd"] Nov 01 07:11:29.514375 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:11:29.514335 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff14bf0d-49ba-43b1-a64c-61e3482983b8" path="/var/lib/kubelet/pods/ff14bf0d-49ba-43b1-a64c-61e3482983b8/volumes" Nov 01 07:11:39.976617 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:11:39.976520 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:12:09.972655 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:12:09.972611 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:12:39.976048 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:12:39.976005 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:13:09.972271 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:13:09.972235 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:13:19.046766 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:13:19.046730 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5771ebbb-qwc29"] Nov 01 07:13:19.097504 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:13:19.097448 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5771ebbb-qwc29"] Nov 01 07:13:19.514075 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:13:19.514040 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b39ae85-fa77-4034-bde6-ecadedab7242" path="/var/lib/kubelet/pods/3b39ae85-fa77-4034-bde6-ecadedab7242/volumes" Nov 01 07:13:39.972764 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:13:39.972719 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:14:08.047046 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:14:08.047008 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88e79a77-vkxc8"] Nov 01 07:14:08.097636 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:14:08.097601 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88e79a77-vkxc8"] Nov 01 07:14:09.514146 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:14:09.514111 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d8915e4-9ce5-41ee-92a3-9f6641a3c285" path="/var/lib/kubelet/pods/0d8915e4-9ce5-41ee-92a3-9f6641a3c285/volumes" Nov 01 07:14:09.972948 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:14:09.972863 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:14:24.533599 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:14:24.533563 5040 scope.go:122] "RemoveContainer" containerID="af345e8cf86bb239bb4c911020a66799b5c5fae087b48cba54ce7c7d818e5301" Nov 01 07:14:39.972046 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:14:39.972003 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:14:49.861345 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:14:49.860651 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tnrtq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 07:14:49.861345 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:14:49.860745 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" podUID="09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 07:15:09.972215 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:15:09.972178 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:15:14.388917 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:14.388879 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xlx7r"] Nov 01 07:15:14.389347 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:14.389299 5040 cpu_manager.go:414] "containerMap: removing container" podUID="ff14bf0d-49ba-43b1-a64c-61e3482983b8" containerName="main" Nov 01 07:15:14.389347 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:14.389311 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff14bf0d-49ba-43b1-a64c-61e3482983b8" containerName="main" Nov 01 07:15:14.389435 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:14.389385 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff14bf0d-49ba-43b1-a64c-61e3482983b8" containerName="main" Nov 01 07:15:14.393334 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:14.393318 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xlx7r" Nov 01 07:15:14.497723 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:14.497689 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 07:15:14.497916 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:14.497728 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 07:15:14.497916 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:14.497729 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 07:15:14.497916 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:14.497826 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cf2w\"" Nov 01 07:15:14.500117 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:14.500093 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xlx7r"] Nov 01 07:15:14.501555 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:14.501523 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97dwb\" (UniqueName: \"kubernetes.io/projected/db5e6d00-27b4-4587-a485-0829049c51c7-kube-api-access-97dwb\") pod \"maas-api-revoke-sub-c1e271a9-xlx7r\" (UID: \"db5e6d00-27b4-4587-a485-0829049c51c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xlx7r" Nov 01 07:15:14.501659 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:14.501631 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db5e6d00-27b4-4587-a485-0829049c51c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-xlx7r\" (UID: \"db5e6d00-27b4-4587-a485-0829049c51c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xlx7r" Nov 01 07:15:14.501709 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:14.501695 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db5e6d00-27b4-4587-a485-0829049c51c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-xlx7r\" (UID: \"db5e6d00-27b4-4587-a485-0829049c51c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xlx7r" Nov 01 07:15:14.602697 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:14.602658 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97dwb\" (UniqueName: \"kubernetes.io/projected/db5e6d00-27b4-4587-a485-0829049c51c7-kube-api-access-97dwb\") pod \"maas-api-revoke-sub-c1e271a9-xlx7r\" (UID: \"db5e6d00-27b4-4587-a485-0829049c51c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xlx7r" Nov 01 07:15:14.602875 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:14.602819 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db5e6d00-27b4-4587-a485-0829049c51c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-xlx7r\" (UID: \"db5e6d00-27b4-4587-a485-0829049c51c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xlx7r" Nov 01 07:15:14.602932 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:14.602914 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db5e6d00-27b4-4587-a485-0829049c51c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-xlx7r\" (UID: \"db5e6d00-27b4-4587-a485-0829049c51c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xlx7r" Nov 01 07:15:14.603527 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:14.603472 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db5e6d00-27b4-4587-a485-0829049c51c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-xlx7r\" (UID: \"db5e6d00-27b4-4587-a485-0829049c51c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xlx7r" Nov 01 07:15:14.605228 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:14.605202 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db5e6d00-27b4-4587-a485-0829049c51c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-xlx7r\" (UID: \"db5e6d00-27b4-4587-a485-0829049c51c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xlx7r" Nov 01 07:15:14.665018 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:14.664949 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97dwb\" (UniqueName: \"kubernetes.io/projected/db5e6d00-27b4-4587-a485-0829049c51c7-kube-api-access-97dwb\") pod \"maas-api-revoke-sub-c1e271a9-xlx7r\" (UID: \"db5e6d00-27b4-4587-a485-0829049c51c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xlx7r" Nov 01 07:15:14.706815 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:14.706777 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xlx7r" Nov 01 07:15:15.080925 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:15.080893 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xlx7r"] Nov 01 07:15:15.082718 ip-10-0-143-195 kubenswrapper[5040]: W1101 07:15:15.082688 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb5e6d00_27b4_4587_a485_0829049c51c7.slice/crio-bc9f905825e6b42199b05836150bc63e32b165d999415de5a9056874ab171e8a WatchSource:0}: Error finding container bc9f905825e6b42199b05836150bc63e32b165d999415de5a9056874ab171e8a: Status 404 returned error can't find the container with id bc9f905825e6b42199b05836150bc63e32b165d999415de5a9056874ab171e8a Nov 01 07:15:15.551974 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:15.551934 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xlx7r" event={"ID":"db5e6d00-27b4-4587-a485-0829049c51c7","Type":"ContainerStarted","Data":"5011ffbf1aa6adb1124bdf596c60be99fb4962f0bcd41d88d57ce8b614610d57"} Nov 01 07:15:15.552459 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:15.551998 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xlx7r" event={"ID":"db5e6d00-27b4-4587-a485-0829049c51c7","Type":"ContainerStarted","Data":"bc9f905825e6b42199b05836150bc63e32b165d999415de5a9056874ab171e8a"} Nov 01 07:15:15.670079 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:15.669976 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xlx7r" podStartSLOduration=1.669963515 podStartE2EDuration="1.669963515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:15:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:15:15.668701702 +0000 UTC m=+5394.323818869" watchObservedRunningTime="2026-11-01 07:15:15.669963515 +0000 UTC m=+5394.325080676" Nov 01 07:15:16.558533 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:16.558498 5040 generic.go:356] "Generic (PLEG): container finished" podID="db5e6d00-27b4-4587-a485-0829049c51c7" containerID="5011ffbf1aa6adb1124bdf596c60be99fb4962f0bcd41d88d57ce8b614610d57" exitCode=0 Nov 01 07:15:16.558933 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:16.558588 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xlx7r" event={"ID":"db5e6d00-27b4-4587-a485-0829049c51c7","Type":"ContainerDied","Data":"5011ffbf1aa6adb1124bdf596c60be99fb4962f0bcd41d88d57ce8b614610d57"} Nov 01 07:15:17.707329 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:17.707301 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xlx7r" Nov 01 07:15:17.830656 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:17.830579 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db5e6d00-27b4-4587-a485-0829049c51c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db5e6d00-27b4-4587-a485-0829049c51c7-maas-api-xks-ca\") pod \"db5e6d00-27b4-4587-a485-0829049c51c7\" (UID: \"db5e6d00-27b4-4587-a485-0829049c51c7\") " Nov 01 07:15:17.830656 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:17.830641 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db5e6d00-27b4-4587-a485-0829049c51c7-kube-api-access-97dwb\" (UniqueName: \"kubernetes.io/projected/db5e6d00-27b4-4587-a485-0829049c51c7-kube-api-access-97dwb\") pod \"db5e6d00-27b4-4587-a485-0829049c51c7\" (UID: \"db5e6d00-27b4-4587-a485-0829049c51c7\") " Nov 01 07:15:17.830900 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:17.830671 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db5e6d00-27b4-4587-a485-0829049c51c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db5e6d00-27b4-4587-a485-0829049c51c7-maas-api-openshift-ca\") pod \"db5e6d00-27b4-4587-a485-0829049c51c7\" (UID: \"db5e6d00-27b4-4587-a485-0829049c51c7\") " Nov 01 07:15:17.831329 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:17.831281 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db5e6d00-27b4-4587-a485-0829049c51c7-maas-api-openshift-ca" pod "db5e6d00-27b4-4587-a485-0829049c51c7" (UID: "db5e6d00-27b4-4587-a485-0829049c51c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:15:17.832821 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:17.832795 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db5e6d00-27b4-4587-a485-0829049c51c7-maas-api-xks-ca" pod "db5e6d00-27b4-4587-a485-0829049c51c7" (UID: "db5e6d00-27b4-4587-a485-0829049c51c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:15:17.833224 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:17.833208 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db5e6d00-27b4-4587-a485-0829049c51c7-kube-api-access-97dwb" pod "db5e6d00-27b4-4587-a485-0829049c51c7" (UID: "db5e6d00-27b4-4587-a485-0829049c51c7"). InnerVolumeSpecName "kube-api-access-97dwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:15:17.932048 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:17.932005 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db5e6d00-27b4-4587-a485-0829049c51c7-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:15:17.932048 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:17.932047 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97dwb\" (UniqueName: \"kubernetes.io/projected/db5e6d00-27b4-4587-a485-0829049c51c7-kube-api-access-97dwb\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:15:17.932048 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:17.932058 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db5e6d00-27b4-4587-a485-0829049c51c7-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:15:18.571153 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:18.571119 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xlx7r" Nov 01 07:15:18.571340 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:18.571152 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xlx7r" event={"ID":"db5e6d00-27b4-4587-a485-0829049c51c7","Type":"ContainerDied","Data":"bc9f905825e6b42199b05836150bc63e32b165d999415de5a9056874ab171e8a"} Nov 01 07:15:18.571340 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:18.571198 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc9f905825e6b42199b05836150bc63e32b165d999415de5a9056874ab171e8a" Nov 01 07:15:22.441630 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:22.441507 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 07:15:22.460806 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:22.460780 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 07:15:24.555310 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:15:24.555279 5040 scope.go:122] "RemoveContainer" containerID="adeea783356a0d850f3532b2b4d3b076bb9a7db15efa212ad7412b3201664066" Nov 01 07:15:39.972071 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:15:39.972025 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:16:09.972287 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:16:09.972247 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:16:39.990105 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:16:39.990054 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:17:09.971451 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:17:09.971407 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:17:39.972459 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:17:39.972418 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:18:09.972683 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:18:09.972643 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:18:39.972468 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:18:39.972432 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:19:09.972459 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:19:09.972414 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:19:39.972197 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:19:39.972148 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:20:09.972038 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:20:09.971998 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:20:18.047092 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:20:18.047055 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xlx7r"] Nov 01 07:20:18.097415 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:20:18.097379 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xlx7r"] Nov 01 07:20:19.513391 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:20:19.513358 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db5e6d00-27b4-4587-a485-0829049c51c7" path="/var/lib/kubelet/pods/db5e6d00-27b4-4587-a485-0829049c51c7/volumes" Nov 01 07:20:22.479423 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:20:22.479307 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 07:20:22.497646 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:20:22.497621 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 07:20:39.972911 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:20:39.972868 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:21:06.407501 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:06.406633 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-f895b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 07:21:06.407501 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:06.406735 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f895b" podUID="2b85392e-5dd8-4539-93ac-e659fb9cdf2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 07:21:09.972687 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:21:09.972651 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:21:24.622753 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:24.622715 5040 scope.go:122] "RemoveContainer" containerID="5011ffbf1aa6adb1124bdf596c60be99fb4962f0bcd41d88d57ce8b614610d57" Nov 01 07:21:39.972753 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:21:39.972711 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:21:46.524849 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:46.524804 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc"] Nov 01 07:21:46.525417 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:46.525342 5040 cpu_manager.go:414] "containerMap: removing container" podUID="db5e6d00-27b4-4587-a485-0829049c51c7" containerName="revoke-keys" Nov 01 07:21:46.525417 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:46.525366 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db5e6d00-27b4-4587-a485-0829049c51c7" containerName="revoke-keys" Nov 01 07:21:46.525567 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:46.525512 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="db5e6d00-27b4-4587-a485-0829049c51c7" containerName="revoke-keys" Nov 01 07:21:46.528684 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:46.528663 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc" Nov 01 07:21:46.586151 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:46.586122 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc"] Nov 01 07:21:46.586566 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:46.586538 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-c6c6bf\"/\"openshift-service-ca.crt\"" Nov 01 07:21:46.586724 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:46.586631 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-c6c6bf\"/\"kube-root-ca.crt\"" Nov 01 07:21:46.634459 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:46.634427 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-c6c6bf\"/\"default-dockercfg-2f6hk\"" Nov 01 07:21:46.634636 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:46.634544 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-c6c6bf\"/\"simulator-single-main-c6c6bf-kserve-self-signed-certs\"" Nov 01 07:21:46.642753 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:46.642730 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/350a8f13-72ca-4d67-bd13-23f594136433-home\") pod \"simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc\" (UID: \"350a8f13-72ca-4d67-bd13-23f594136433\") " pod="e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc" Nov 01 07:21:46.642873 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:46.642781 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/350a8f13-72ca-4d67-bd13-23f594136433-tmp-dir\") pod \"simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc\" (UID: \"350a8f13-72ca-4d67-bd13-23f594136433\") " pod="e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc" Nov 01 07:21:46.643108 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:46.643084 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/350a8f13-72ca-4d67-bd13-23f594136433-model-cache\") pod \"simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc\" (UID: \"350a8f13-72ca-4d67-bd13-23f594136433\") " pod="e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc" Nov 01 07:21:46.643215 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:46.643171 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/350a8f13-72ca-4d67-bd13-23f594136433-tls-certs\") pod \"simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc\" (UID: \"350a8f13-72ca-4d67-bd13-23f594136433\") " pod="e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc" Nov 01 07:21:46.643278 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:46.643249 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll68n\" (UniqueName: \"kubernetes.io/projected/350a8f13-72ca-4d67-bd13-23f594136433-kube-api-access-ll68n\") pod \"simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc\" (UID: \"350a8f13-72ca-4d67-bd13-23f594136433\") " pod="e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc" Nov 01 07:21:46.643391 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:46.643372 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/350a8f13-72ca-4d67-bd13-23f594136433-dshm\") pod \"simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc\" (UID: \"350a8f13-72ca-4d67-bd13-23f594136433\") " pod="e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc" Nov 01 07:21:46.744929 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:46.744891 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/350a8f13-72ca-4d67-bd13-23f594136433-model-cache\") pod \"simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc\" (UID: \"350a8f13-72ca-4d67-bd13-23f594136433\") " pod="e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc" Nov 01 07:21:46.745161 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:46.744950 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/350a8f13-72ca-4d67-bd13-23f594136433-tls-certs\") pod \"simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc\" (UID: \"350a8f13-72ca-4d67-bd13-23f594136433\") " pod="e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc" Nov 01 07:21:46.745161 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:46.744988 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll68n\" (UniqueName: \"kubernetes.io/projected/350a8f13-72ca-4d67-bd13-23f594136433-kube-api-access-ll68n\") pod \"simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc\" (UID: \"350a8f13-72ca-4d67-bd13-23f594136433\") " pod="e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc" Nov 01 07:21:46.745161 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:46.745017 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/350a8f13-72ca-4d67-bd13-23f594136433-dshm\") pod \"simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc\" (UID: \"350a8f13-72ca-4d67-bd13-23f594136433\") " pod="e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc" Nov 01 07:21:46.745161 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:46.745044 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/350a8f13-72ca-4d67-bd13-23f594136433-home\") pod \"simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc\" (UID: \"350a8f13-72ca-4d67-bd13-23f594136433\") " pod="e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc" Nov 01 07:21:46.745364 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:46.745242 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/350a8f13-72ca-4d67-bd13-23f594136433-tmp-dir\") pod \"simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc\" (UID: \"350a8f13-72ca-4d67-bd13-23f594136433\") " pod="e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc" Nov 01 07:21:46.745782 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:46.745750 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/350a8f13-72ca-4d67-bd13-23f594136433-model-cache\") pod \"simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc\" (UID: \"350a8f13-72ca-4d67-bd13-23f594136433\") " pod="e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc" Nov 01 07:21:46.745943 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:46.745777 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/350a8f13-72ca-4d67-bd13-23f594136433-home\") pod \"simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc\" (UID: \"350a8f13-72ca-4d67-bd13-23f594136433\") " pod="e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc" Nov 01 07:21:46.745943 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:46.745829 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/350a8f13-72ca-4d67-bd13-23f594136433-tmp-dir\") pod \"simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc\" (UID: \"350a8f13-72ca-4d67-bd13-23f594136433\") " pod="e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc" Nov 01 07:21:46.747752 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:46.747720 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/350a8f13-72ca-4d67-bd13-23f594136433-dshm\") pod \"simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc\" (UID: \"350a8f13-72ca-4d67-bd13-23f594136433\") " pod="e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc" Nov 01 07:21:46.747974 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:46.747954 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/350a8f13-72ca-4d67-bd13-23f594136433-tls-certs\") pod \"simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc\" (UID: \"350a8f13-72ca-4d67-bd13-23f594136433\") " pod="e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc" Nov 01 07:21:46.807017 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:46.806946 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll68n\" (UniqueName: \"kubernetes.io/projected/350a8f13-72ca-4d67-bd13-23f594136433-kube-api-access-ll68n\") pod \"simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc\" (UID: \"350a8f13-72ca-4d67-bd13-23f594136433\") " pod="e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc" Nov 01 07:21:46.839951 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:46.839916 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc" Nov 01 07:21:47.217683 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:47.217658 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc"] Nov 01 07:21:47.219562 ip-10-0-143-195 kubenswrapper[5040]: W1101 07:21:47.219534 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod350a8f13_72ca_4d67_bd13_23f594136433.slice/crio-ed8256a211f2bfd07afadcd657722d4006ab0bbf0c02d78b7085d7179bc4b2a7 WatchSource:0}: Error finding container ed8256a211f2bfd07afadcd657722d4006ab0bbf0c02d78b7085d7179bc4b2a7: Status 404 returned error can't find the container with id ed8256a211f2bfd07afadcd657722d4006ab0bbf0c02d78b7085d7179bc4b2a7 Nov 01 07:21:47.867273 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:47.867228 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc" event={"ID":"350a8f13-72ca-4d67-bd13-23f594136433","Type":"ContainerStarted","Data":"d6b6877eaaf86eba4db3351eef97e6fb98b8bf196a09e2c44a24c8b825c60d76"} Nov 01 07:21:47.867736 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:47.867286 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc" event={"ID":"350a8f13-72ca-4d67-bd13-23f594136433","Type":"ContainerStarted","Data":"ed8256a211f2bfd07afadcd657722d4006ab0bbf0c02d78b7085d7179bc4b2a7"} Nov 01 07:21:47.986182 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:47.986130 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc" podStartSLOduration=1.986116121 podStartE2EDuration="1.986116121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:21:47.984367886 +0000 UTC m=+5786.639485051" watchObservedRunningTime="2026-11-01 07:21:47.986116121 +0000 UTC m=+5786.641233288" Nov 01 07:21:48.840492 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:48.840454 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc" Nov 01 07:21:56.840083 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:56.840039 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc" Nov 01 07:21:56.852316 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:56.852278 5040 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc" Nov 01 07:21:56.925930 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:21:56.925898 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc" Nov 01 07:22:09.972136 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:22:09.972090 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:22:30.493622 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:22:30.493583 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c1e9675-tllgm"] Nov 01 07:22:30.497367 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:22:30.497341 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c1e9675-tllgm" Nov 01 07:22:30.554687 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:22:30.554657 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c1e9675-tllgm"] Nov 01 07:22:30.602257 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:22:30.602218 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 07:22:30.602257 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:22:30.602243 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 07:22:30.602452 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:22:30.602305 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 07:22:30.602452 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:22:30.602321 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cf2w\"" Nov 01 07:22:30.665289 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:22:30.665253 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c850a03-796c-46df-8085-896ed87eb85c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c1e9675-tllgm\" (UID: \"2c850a03-796c-46df-8085-896ed87eb85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c1e9675-tllgm" Nov 01 07:22:30.665289 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:22:30.665290 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c850a03-796c-46df-8085-896ed87eb85c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c1e9675-tllgm\" (UID: \"2c850a03-796c-46df-8085-896ed87eb85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c1e9675-tllgm" Nov 01 07:22:30.665518 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:22:30.665318 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfrt8\" (UniqueName: \"kubernetes.io/projected/2c850a03-796c-46df-8085-896ed87eb85c-kube-api-access-lfrt8\") pod \"maas-api-revoke-sub-3c1e9675-tllgm\" (UID: \"2c850a03-796c-46df-8085-896ed87eb85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c1e9675-tllgm" Nov 01 07:22:30.766276 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:22:30.766237 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfrt8\" (UniqueName: \"kubernetes.io/projected/2c850a03-796c-46df-8085-896ed87eb85c-kube-api-access-lfrt8\") pod \"maas-api-revoke-sub-3c1e9675-tllgm\" (UID: \"2c850a03-796c-46df-8085-896ed87eb85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c1e9675-tllgm" Nov 01 07:22:30.766515 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:22:30.766396 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c850a03-796c-46df-8085-896ed87eb85c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c1e9675-tllgm\" (UID: \"2c850a03-796c-46df-8085-896ed87eb85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c1e9675-tllgm" Nov 01 07:22:30.766515 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:22:30.766426 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c850a03-796c-46df-8085-896ed87eb85c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c1e9675-tllgm\" (UID: \"2c850a03-796c-46df-8085-896ed87eb85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c1e9675-tllgm" Nov 01 07:22:30.767113 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:22:30.767087 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c850a03-796c-46df-8085-896ed87eb85c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c1e9675-tllgm\" (UID: \"2c850a03-796c-46df-8085-896ed87eb85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c1e9675-tllgm" Nov 01 07:22:30.768827 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:22:30.768807 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c850a03-796c-46df-8085-896ed87eb85c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c1e9675-tllgm\" (UID: \"2c850a03-796c-46df-8085-896ed87eb85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c1e9675-tllgm" Nov 01 07:22:30.828138 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:22:30.828103 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfrt8\" (UniqueName: \"kubernetes.io/projected/2c850a03-796c-46df-8085-896ed87eb85c-kube-api-access-lfrt8\") pod \"maas-api-revoke-sub-3c1e9675-tllgm\" (UID: \"2c850a03-796c-46df-8085-896ed87eb85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c1e9675-tllgm" Nov 01 07:22:31.111247 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:22:31.111167 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c1e9675-tllgm" Nov 01 07:22:31.483303 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:22:31.483273 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c1e9675-tllgm"] Nov 01 07:22:31.484814 ip-10-0-143-195 kubenswrapper[5040]: W1101 07:22:31.484782 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c850a03_796c_46df_8085_896ed87eb85c.slice/crio-e906503dcf4d97394e0af6467fabd7995e08c83165deb19dd075f03435a7d9f4 WatchSource:0}: Error finding container e906503dcf4d97394e0af6467fabd7995e08c83165deb19dd075f03435a7d9f4: Status 404 returned error can't find the container with id e906503dcf4d97394e0af6467fabd7995e08c83165deb19dd075f03435a7d9f4 Nov 01 07:22:32.104906 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:22:32.104815 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c1e9675-tllgm" event={"ID":"2c850a03-796c-46df-8085-896ed87eb85c","Type":"ContainerStarted","Data":"2c0488de18069872cd29418cee3cf003ddc06da1a4849575eb81bef24fb74deb"} Nov 01 07:22:32.104906 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:22:32.104874 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c1e9675-tllgm" event={"ID":"2c850a03-796c-46df-8085-896ed87eb85c","Type":"ContainerStarted","Data":"e906503dcf4d97394e0af6467fabd7995e08c83165deb19dd075f03435a7d9f4"} Nov 01 07:22:32.222923 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:22:32.222869 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c1e9675-tllgm" podStartSLOduration=2.222855852 podStartE2EDuration="2.222855852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:22:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:22:32.220988552 +0000 UTC m=+5830.876105719" watchObservedRunningTime="2026-11-01 07:22:32.222855852 +0000 UTC m=+5830.877973018" Nov 01 07:22:33.111774 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:22:33.111738 5040 generic.go:356] "Generic (PLEG): container finished" podID="2c850a03-796c-46df-8085-896ed87eb85c" containerID="2c0488de18069872cd29418cee3cf003ddc06da1a4849575eb81bef24fb74deb" exitCode=0 Nov 01 07:22:33.112173 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:22:33.111818 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c1e9675-tllgm" event={"ID":"2c850a03-796c-46df-8085-896ed87eb85c","Type":"ContainerDied","Data":"2c0488de18069872cd29418cee3cf003ddc06da1a4849575eb81bef24fb74deb"} Nov 01 07:22:34.262142 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:22:34.262119 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c1e9675-tllgm" Nov 01 07:22:34.400254 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:22:34.400173 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c850a03-796c-46df-8085-896ed87eb85c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c850a03-796c-46df-8085-896ed87eb85c-maas-api-openshift-ca\") pod \"2c850a03-796c-46df-8085-896ed87eb85c\" (UID: \"2c850a03-796c-46df-8085-896ed87eb85c\") " Nov 01 07:22:34.400440 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:22:34.400269 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c850a03-796c-46df-8085-896ed87eb85c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c850a03-796c-46df-8085-896ed87eb85c-maas-api-xks-ca\") pod \"2c850a03-796c-46df-8085-896ed87eb85c\" (UID: \"2c850a03-796c-46df-8085-896ed87eb85c\") " Nov 01 07:22:34.400440 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:22:34.400319 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c850a03-796c-46df-8085-896ed87eb85c-kube-api-access-lfrt8\" (UniqueName: \"kubernetes.io/projected/2c850a03-796c-46df-8085-896ed87eb85c-kube-api-access-lfrt8\") pod \"2c850a03-796c-46df-8085-896ed87eb85c\" (UID: \"2c850a03-796c-46df-8085-896ed87eb85c\") " Nov 01 07:22:34.400653 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:22:34.400629 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c850a03-796c-46df-8085-896ed87eb85c-maas-api-openshift-ca" pod "2c850a03-796c-46df-8085-896ed87eb85c" (UID: "2c850a03-796c-46df-8085-896ed87eb85c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:22:34.402354 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:22:34.402332 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c850a03-796c-46df-8085-896ed87eb85c-kube-api-access-lfrt8" pod "2c850a03-796c-46df-8085-896ed87eb85c" (UID: "2c850a03-796c-46df-8085-896ed87eb85c"). InnerVolumeSpecName "kube-api-access-lfrt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:22:34.402423 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:22:34.402333 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c850a03-796c-46df-8085-896ed87eb85c-maas-api-xks-ca" pod "2c850a03-796c-46df-8085-896ed87eb85c" (UID: "2c850a03-796c-46df-8085-896ed87eb85c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:22:34.501088 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:22:34.501048 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c850a03-796c-46df-8085-896ed87eb85c-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:22:34.501088 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:22:34.501078 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfrt8\" (UniqueName: \"kubernetes.io/projected/2c850a03-796c-46df-8085-896ed87eb85c-kube-api-access-lfrt8\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:22:34.501088 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:22:34.501096 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c850a03-796c-46df-8085-896ed87eb85c-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:22:35.125990 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:22:35.125962 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c1e9675-tllgm" Nov 01 07:22:35.126172 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:22:35.125997 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c1e9675-tllgm" event={"ID":"2c850a03-796c-46df-8085-896ed87eb85c","Type":"ContainerDied","Data":"e906503dcf4d97394e0af6467fabd7995e08c83165deb19dd075f03435a7d9f4"} Nov 01 07:22:35.126172 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:22:35.126049 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e906503dcf4d97394e0af6467fabd7995e08c83165deb19dd075f03435a7d9f4" Nov 01 07:22:39.972501 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:22:39.972447 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:22:47.152925 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:22:47.152886 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc"] Nov 01 07:23:02.173004 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:23:02.172944 5040 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc" podUID="350a8f13-72ca-4d67-bd13-23f594136433" containerName="main" containerID="cri-o://d6b6877eaaf86eba4db3351eef97e6fb98b8bf196a09e2c44a24c8b825c60d76" gracePeriod=45 Nov 01 07:23:02.425328 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:23:02.425263 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc" Nov 01 07:23:02.557688 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:23:02.557659 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/350a8f13-72ca-4d67-bd13-23f594136433-dshm\" (UniqueName: \"kubernetes.io/empty-dir/350a8f13-72ca-4d67-bd13-23f594136433-dshm\") pod \"350a8f13-72ca-4d67-bd13-23f594136433\" (UID: \"350a8f13-72ca-4d67-bd13-23f594136433\") " Nov 01 07:23:02.557870 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:23:02.557737 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/350a8f13-72ca-4d67-bd13-23f594136433-tls-certs\" (UniqueName: \"kubernetes.io/secret/350a8f13-72ca-4d67-bd13-23f594136433-tls-certs\") pod \"350a8f13-72ca-4d67-bd13-23f594136433\" (UID: \"350a8f13-72ca-4d67-bd13-23f594136433\") " Nov 01 07:23:02.557870 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:23:02.557779 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/350a8f13-72ca-4d67-bd13-23f594136433-kube-api-access-ll68n\" (UniqueName: \"kubernetes.io/projected/350a8f13-72ca-4d67-bd13-23f594136433-kube-api-access-ll68n\") pod \"350a8f13-72ca-4d67-bd13-23f594136433\" (UID: \"350a8f13-72ca-4d67-bd13-23f594136433\") " Nov 01 07:23:02.557870 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:23:02.557810 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/350a8f13-72ca-4d67-bd13-23f594136433-home\" (UniqueName: \"kubernetes.io/empty-dir/350a8f13-72ca-4d67-bd13-23f594136433-home\") pod \"350a8f13-72ca-4d67-bd13-23f594136433\" (UID: \"350a8f13-72ca-4d67-bd13-23f594136433\") " Nov 01 07:23:02.558063 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:23:02.557914 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/350a8f13-72ca-4d67-bd13-23f594136433-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/350a8f13-72ca-4d67-bd13-23f594136433-model-cache\") pod \"350a8f13-72ca-4d67-bd13-23f594136433\" (UID: \"350a8f13-72ca-4d67-bd13-23f594136433\") " Nov 01 07:23:02.558063 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:23:02.557995 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/350a8f13-72ca-4d67-bd13-23f594136433-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/350a8f13-72ca-4d67-bd13-23f594136433-tmp-dir\") pod \"350a8f13-72ca-4d67-bd13-23f594136433\" (UID: \"350a8f13-72ca-4d67-bd13-23f594136433\") " Nov 01 07:23:02.558187 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:23:02.558090 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/350a8f13-72ca-4d67-bd13-23f594136433-home" pod "350a8f13-72ca-4d67-bd13-23f594136433" (UID: "350a8f13-72ca-4d67-bd13-23f594136433"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:23:02.558421 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:23:02.558244 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/350a8f13-72ca-4d67-bd13-23f594136433-model-cache" pod "350a8f13-72ca-4d67-bd13-23f594136433" (UID: "350a8f13-72ca-4d67-bd13-23f594136433"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:23:02.558421 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:23:02.558339 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/350a8f13-72ca-4d67-bd13-23f594136433-tmp-dir" pod "350a8f13-72ca-4d67-bd13-23f594136433" (UID: "350a8f13-72ca-4d67-bd13-23f594136433"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:23:02.558421 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:23:02.558384 5040 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/350a8f13-72ca-4d67-bd13-23f594136433-home\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:23:02.558421 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:23:02.558398 5040 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/350a8f13-72ca-4d67-bd13-23f594136433-model-cache\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:23:02.560440 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:23:02.560412 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/350a8f13-72ca-4d67-bd13-23f594136433-tls-certs" pod "350a8f13-72ca-4d67-bd13-23f594136433" (UID: "350a8f13-72ca-4d67-bd13-23f594136433"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:23:02.560563 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:23:02.560517 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/350a8f13-72ca-4d67-bd13-23f594136433-dshm" pod "350a8f13-72ca-4d67-bd13-23f594136433" (UID: "350a8f13-72ca-4d67-bd13-23f594136433"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:23:02.560669 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:23:02.560645 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/350a8f13-72ca-4d67-bd13-23f594136433-kube-api-access-ll68n" pod "350a8f13-72ca-4d67-bd13-23f594136433" (UID: "350a8f13-72ca-4d67-bd13-23f594136433"). InnerVolumeSpecName "kube-api-access-ll68n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:23:02.659338 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:23:02.659303 5040 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/350a8f13-72ca-4d67-bd13-23f594136433-tls-certs\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:23:02.659338 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:23:02.659333 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll68n\" (UniqueName: \"kubernetes.io/projected/350a8f13-72ca-4d67-bd13-23f594136433-kube-api-access-ll68n\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:23:02.659338 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:23:02.659344 5040 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/350a8f13-72ca-4d67-bd13-23f594136433-tmp-dir\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:23:02.659623 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:23:02.659352 5040 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/350a8f13-72ca-4d67-bd13-23f594136433-dshm\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:23:03.271975 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:23:03.271932 5040 generic.go:356] "Generic (PLEG): container finished" podID="350a8f13-72ca-4d67-bd13-23f594136433" containerID="d6b6877eaaf86eba4db3351eef97e6fb98b8bf196a09e2c44a24c8b825c60d76" exitCode=0 Nov 01 07:23:03.272512 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:23:03.271997 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc" Nov 01 07:23:03.272512 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:23:03.272022 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc" event={"ID":"350a8f13-72ca-4d67-bd13-23f594136433","Type":"ContainerDied","Data":"d6b6877eaaf86eba4db3351eef97e6fb98b8bf196a09e2c44a24c8b825c60d76"} Nov 01 07:23:03.272512 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:23:03.272083 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc" event={"ID":"350a8f13-72ca-4d67-bd13-23f594136433","Type":"ContainerDied","Data":"ed8256a211f2bfd07afadcd657722d4006ab0bbf0c02d78b7085d7179bc4b2a7"} Nov 01 07:23:03.272512 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:23:03.272107 5040 scope.go:122] "RemoveContainer" containerID="d6b6877eaaf86eba4db3351eef97e6fb98b8bf196a09e2c44a24c8b825c60d76" Nov 01 07:23:03.282777 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:23:03.282757 5040 scope.go:122] "RemoveContainer" containerID="d6b6877eaaf86eba4db3351eef97e6fb98b8bf196a09e2c44a24c8b825c60d76" Nov 01 07:23:03.283042 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:23:03.283022 5040 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6b6877eaaf86eba4db3351eef97e6fb98b8bf196a09e2c44a24c8b825c60d76\": container with ID starting with d6b6877eaaf86eba4db3351eef97e6fb98b8bf196a09e2c44a24c8b825c60d76 not found: ID does not exist" containerID="d6b6877eaaf86eba4db3351eef97e6fb98b8bf196a09e2c44a24c8b825c60d76" Nov 01 07:23:03.283103 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:23:03.283055 5040 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6b6877eaaf86eba4db3351eef97e6fb98b8bf196a09e2c44a24c8b825c60d76"} err="failed to get container status \"d6b6877eaaf86eba4db3351eef97e6fb98b8bf196a09e2c44a24c8b825c60d76\": rpc error: code = NotFound desc = could not find container \"d6b6877eaaf86eba4db3351eef97e6fb98b8bf196a09e2c44a24c8b825c60d76\": container with ID starting with d6b6877eaaf86eba4db3351eef97e6fb98b8bf196a09e2c44a24c8b825c60d76 not found: ID does not exist" Nov 01 07:23:03.441027 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:23:03.440989 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc"] Nov 01 07:23:03.491848 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:23:03.491813 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-c6c6bf/simulator-single-main-c6c6bf-kserve-54b4b8645d-c8hgc"] Nov 01 07:23:03.513841 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:23:03.513808 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="350a8f13-72ca-4d67-bd13-23f594136433" path="/var/lib/kubelet/pods/350a8f13-72ca-4d67-bd13-23f594136433/volumes" Nov 01 07:23:09.972618 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:23:09.972577 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:23:39.972419 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:23:39.972375 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:24:09.972799 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:24:09.972756 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:24:19.862636 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:24:19.862587 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tnrtq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 07:24:19.863383 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:24:19.863342 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" podUID="09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 07:24:39.972672 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:24:39.972633 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:25:04.862737 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:04.862684 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tnrtq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 07:25:04.863955 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:04.863916 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" podUID="09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 07:25:09.106691 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:09.106652 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7782503-m9dz7"] Nov 01 07:25:09.107087 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:09.107036 5040 cpu_manager.go:414] "containerMap: removing container" podUID="2c850a03-796c-46df-8085-896ed87eb85c" containerName="revoke-keys" Nov 01 07:25:09.107087 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:09.107048 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c850a03-796c-46df-8085-896ed87eb85c" containerName="revoke-keys" Nov 01 07:25:09.107087 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:09.107079 5040 cpu_manager.go:414] "containerMap: removing container" podUID="350a8f13-72ca-4d67-bd13-23f594136433" containerName="main" Nov 01 07:25:09.107087 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:09.107084 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="350a8f13-72ca-4d67-bd13-23f594136433" containerName="main" Nov 01 07:25:09.107232 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:09.107151 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="350a8f13-72ca-4d67-bd13-23f594136433" containerName="main" Nov 01 07:25:09.107232 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:09.107162 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c850a03-796c-46df-8085-896ed87eb85c" containerName="revoke-keys" Nov 01 07:25:09.110329 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:09.110313 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7782503-m9dz7" Nov 01 07:25:09.165998 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:09.165966 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 07:25:09.168408 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:09.168384 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7782503-m9dz7"] Nov 01 07:25:09.174029 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:09.174005 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6967\" (UniqueName: \"kubernetes.io/projected/336701aa-8260-4998-841a-8b95f48c013d-kube-api-access-f6967\") pod \"maas-api-revoke-sub-f7782503-m9dz7\" (UID: \"336701aa-8260-4998-841a-8b95f48c013d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7782503-m9dz7" Nov 01 07:25:09.174151 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:09.174044 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/336701aa-8260-4998-841a-8b95f48c013d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7782503-m9dz7\" (UID: \"336701aa-8260-4998-841a-8b95f48c013d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7782503-m9dz7" Nov 01 07:25:09.174151 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:09.174078 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/336701aa-8260-4998-841a-8b95f48c013d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7782503-m9dz7\" (UID: \"336701aa-8260-4998-841a-8b95f48c013d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7782503-m9dz7" Nov 01 07:25:09.215966 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:09.215938 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 07:25:09.216115 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:09.215968 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 07:25:09.216115 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:09.215979 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cf2w\"" Nov 01 07:25:09.274591 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:09.274556 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6967\" (UniqueName: \"kubernetes.io/projected/336701aa-8260-4998-841a-8b95f48c013d-kube-api-access-f6967\") pod \"maas-api-revoke-sub-f7782503-m9dz7\" (UID: \"336701aa-8260-4998-841a-8b95f48c013d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7782503-m9dz7" Nov 01 07:25:09.274763 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:09.274627 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/336701aa-8260-4998-841a-8b95f48c013d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7782503-m9dz7\" (UID: \"336701aa-8260-4998-841a-8b95f48c013d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7782503-m9dz7" Nov 01 07:25:09.274763 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:09.274680 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/336701aa-8260-4998-841a-8b95f48c013d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7782503-m9dz7\" (UID: \"336701aa-8260-4998-841a-8b95f48c013d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7782503-m9dz7" Nov 01 07:25:09.275273 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:09.275249 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/336701aa-8260-4998-841a-8b95f48c013d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7782503-m9dz7\" (UID: \"336701aa-8260-4998-841a-8b95f48c013d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7782503-m9dz7" Nov 01 07:25:09.277139 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:09.277104 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/336701aa-8260-4998-841a-8b95f48c013d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7782503-m9dz7\" (UID: \"336701aa-8260-4998-841a-8b95f48c013d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7782503-m9dz7" Nov 01 07:25:09.336638 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:09.336606 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6967\" (UniqueName: \"kubernetes.io/projected/336701aa-8260-4998-841a-8b95f48c013d-kube-api-access-f6967\") pod \"maas-api-revoke-sub-f7782503-m9dz7\" (UID: \"336701aa-8260-4998-841a-8b95f48c013d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7782503-m9dz7" Nov 01 07:25:09.423381 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:09.423299 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7782503-m9dz7" Nov 01 07:25:09.797526 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:09.797491 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7782503-m9dz7"] Nov 01 07:25:09.798857 ip-10-0-143-195 kubenswrapper[5040]: W1101 07:25:09.798832 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod336701aa_8260_4998_841a_8b95f48c013d.slice/crio-58fc768a5fac6f1e596ad3877ae7c962cb5c02985921cd24e48068f519c04333 WatchSource:0}: Error finding container 58fc768a5fac6f1e596ad3877ae7c962cb5c02985921cd24e48068f519c04333: Status 404 returned error can't find the container with id 58fc768a5fac6f1e596ad3877ae7c962cb5c02985921cd24e48068f519c04333 Nov 01 07:25:09.971935 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:25:09.971895 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:25:09.991174 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:09.991140 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7782503-m9dz7" event={"ID":"336701aa-8260-4998-841a-8b95f48c013d","Type":"ContainerStarted","Data":"58fc768a5fac6f1e596ad3877ae7c962cb5c02985921cd24e48068f519c04333"} Nov 01 07:25:10.997727 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:10.997694 5040 generic.go:356] "Generic (PLEG): container finished" podID="336701aa-8260-4998-841a-8b95f48c013d" containerID="c60a3fb1b1d9e2c4c6cae4c5604a9013d89be04c66ede1b3a2583b75cae3fb90" exitCode=0 Nov 01 07:25:10.998136 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:10.997759 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7782503-m9dz7" event={"ID":"336701aa-8260-4998-841a-8b95f48c013d","Type":"ContainerDied","Data":"c60a3fb1b1d9e2c4c6cae4c5604a9013d89be04c66ede1b3a2583b75cae3fb90"} Nov 01 07:25:12.141441 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:12.141413 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7782503-m9dz7" Nov 01 07:25:12.200893 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:12.200848 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/336701aa-8260-4998-841a-8b95f48c013d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/336701aa-8260-4998-841a-8b95f48c013d-maas-api-openshift-ca\") pod \"336701aa-8260-4998-841a-8b95f48c013d\" (UID: \"336701aa-8260-4998-841a-8b95f48c013d\") " Nov 01 07:25:12.201074 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:12.200945 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/336701aa-8260-4998-841a-8b95f48c013d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/336701aa-8260-4998-841a-8b95f48c013d-maas-api-xks-ca\") pod \"336701aa-8260-4998-841a-8b95f48c013d\" (UID: \"336701aa-8260-4998-841a-8b95f48c013d\") " Nov 01 07:25:12.201074 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:12.201000 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/336701aa-8260-4998-841a-8b95f48c013d-kube-api-access-f6967\" (UniqueName: \"kubernetes.io/projected/336701aa-8260-4998-841a-8b95f48c013d-kube-api-access-f6967\") pod \"336701aa-8260-4998-841a-8b95f48c013d\" (UID: \"336701aa-8260-4998-841a-8b95f48c013d\") " Nov 01 07:25:12.201222 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:12.201201 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/336701aa-8260-4998-841a-8b95f48c013d-maas-api-openshift-ca" pod "336701aa-8260-4998-841a-8b95f48c013d" (UID: "336701aa-8260-4998-841a-8b95f48c013d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:25:12.203112 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:12.203089 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/336701aa-8260-4998-841a-8b95f48c013d-kube-api-access-f6967" pod "336701aa-8260-4998-841a-8b95f48c013d" (UID: "336701aa-8260-4998-841a-8b95f48c013d"). InnerVolumeSpecName "kube-api-access-f6967". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:25:12.203226 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:12.203115 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/336701aa-8260-4998-841a-8b95f48c013d-maas-api-xks-ca" pod "336701aa-8260-4998-841a-8b95f48c013d" (UID: "336701aa-8260-4998-841a-8b95f48c013d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:25:12.302236 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:12.302076 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6967\" (UniqueName: \"kubernetes.io/projected/336701aa-8260-4998-841a-8b95f48c013d-kube-api-access-f6967\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:25:12.302236 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:12.302138 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/336701aa-8260-4998-841a-8b95f48c013d-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:25:12.302236 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:12.302154 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/336701aa-8260-4998-841a-8b95f48c013d-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:25:13.010009 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:13.009978 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7782503-m9dz7" Nov 01 07:25:13.010189 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:13.010017 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7782503-m9dz7" event={"ID":"336701aa-8260-4998-841a-8b95f48c013d","Type":"ContainerDied","Data":"58fc768a5fac6f1e596ad3877ae7c962cb5c02985921cd24e48068f519c04333"} Nov 01 07:25:13.010189 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:13.010068 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58fc768a5fac6f1e596ad3877ae7c962cb5c02985921cd24e48068f519c04333" Nov 01 07:25:19.906102 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:19.906062 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c87c717b-7jndl"] Nov 01 07:25:19.906542 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:19.906528 5040 cpu_manager.go:414] "containerMap: removing container" podUID="336701aa-8260-4998-841a-8b95f48c013d" containerName="revoke-keys" Nov 01 07:25:19.906542 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:19.906541 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="336701aa-8260-4998-841a-8b95f48c013d" containerName="revoke-keys" Nov 01 07:25:19.906640 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:19.906620 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="336701aa-8260-4998-841a-8b95f48c013d" containerName="revoke-keys" Nov 01 07:25:19.909970 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:19.909952 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87c717b-7jndl" Nov 01 07:25:19.966562 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:19.966528 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/810e3c9c-47dd-4e5b-a09f-72ba3c675e03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c87c717b-7jndl\" (UID: \"810e3c9c-47dd-4e5b-a09f-72ba3c675e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87c717b-7jndl" Nov 01 07:25:19.966730 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:19.966589 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/810e3c9c-47dd-4e5b-a09f-72ba3c675e03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c87c717b-7jndl\" (UID: \"810e3c9c-47dd-4e5b-a09f-72ba3c675e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87c717b-7jndl" Nov 01 07:25:19.966783 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:19.966743 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnqg2\" (UniqueName: \"kubernetes.io/projected/810e3c9c-47dd-4e5b-a09f-72ba3c675e03-kube-api-access-dnqg2\") pod \"maas-api-revoke-sub-c87c717b-7jndl\" (UID: \"810e3c9c-47dd-4e5b-a09f-72ba3c675e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87c717b-7jndl" Nov 01 07:25:20.014675 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:20.014636 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 07:25:20.014675 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:20.014663 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 07:25:20.014920 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:20.014668 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 07:25:20.014920 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:20.014762 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cf2w\"" Nov 01 07:25:20.017320 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:20.017292 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c87c717b-7jndl"] Nov 01 07:25:20.067931 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:20.067901 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/810e3c9c-47dd-4e5b-a09f-72ba3c675e03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c87c717b-7jndl\" (UID: \"810e3c9c-47dd-4e5b-a09f-72ba3c675e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87c717b-7jndl" Nov 01 07:25:20.068096 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:20.067950 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/810e3c9c-47dd-4e5b-a09f-72ba3c675e03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c87c717b-7jndl\" (UID: \"810e3c9c-47dd-4e5b-a09f-72ba3c675e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87c717b-7jndl" Nov 01 07:25:20.068096 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:20.067992 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnqg2\" (UniqueName: \"kubernetes.io/projected/810e3c9c-47dd-4e5b-a09f-72ba3c675e03-kube-api-access-dnqg2\") pod \"maas-api-revoke-sub-c87c717b-7jndl\" (UID: \"810e3c9c-47dd-4e5b-a09f-72ba3c675e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87c717b-7jndl" Nov 01 07:25:20.068775 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:20.068754 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/810e3c9c-47dd-4e5b-a09f-72ba3c675e03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c87c717b-7jndl\" (UID: \"810e3c9c-47dd-4e5b-a09f-72ba3c675e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87c717b-7jndl" Nov 01 07:25:20.070318 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:20.070295 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/810e3c9c-47dd-4e5b-a09f-72ba3c675e03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c87c717b-7jndl\" (UID: \"810e3c9c-47dd-4e5b-a09f-72ba3c675e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87c717b-7jndl" Nov 01 07:25:20.130181 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:20.130149 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnqg2\" (UniqueName: \"kubernetes.io/projected/810e3c9c-47dd-4e5b-a09f-72ba3c675e03-kube-api-access-dnqg2\") pod \"maas-api-revoke-sub-c87c717b-7jndl\" (UID: \"810e3c9c-47dd-4e5b-a09f-72ba3c675e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87c717b-7jndl" Nov 01 07:25:20.223748 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:20.223673 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87c717b-7jndl" Nov 01 07:25:20.595823 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:20.595792 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c87c717b-7jndl"] Nov 01 07:25:20.597263 ip-10-0-143-195 kubenswrapper[5040]: W1101 07:25:20.597237 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod810e3c9c_47dd_4e5b_a09f_72ba3c675e03.slice/crio-0ed6aa4599a6219a5e783e3a6ec8e9ee81cb7bd132ad309e6d9e7adb9296848c WatchSource:0}: Error finding container 0ed6aa4599a6219a5e783e3a6ec8e9ee81cb7bd132ad309e6d9e7adb9296848c: Status 404 returned error can't find the container with id 0ed6aa4599a6219a5e783e3a6ec8e9ee81cb7bd132ad309e6d9e7adb9296848c Nov 01 07:25:21.054547 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:21.054507 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87c717b-7jndl" event={"ID":"810e3c9c-47dd-4e5b-a09f-72ba3c675e03","Type":"ContainerStarted","Data":"6a8f4895ec073157d4af1709c79d53cbecfa32dec1e1108ebeab52680fa57925"} Nov 01 07:25:21.055049 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:21.054560 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87c717b-7jndl" event={"ID":"810e3c9c-47dd-4e5b-a09f-72ba3c675e03","Type":"ContainerStarted","Data":"0ed6aa4599a6219a5e783e3a6ec8e9ee81cb7bd132ad309e6d9e7adb9296848c"} Nov 01 07:25:21.226536 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:21.226416 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87c717b-7jndl" podStartSLOduration=2.226401224 podStartE2EDuration="2.226401224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:25:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:25:21.22563358 +0000 UTC m=+5999.880750746" watchObservedRunningTime="2026-11-01 07:25:21.226401224 +0000 UTC m=+5999.881518369" Nov 01 07:25:22.064900 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:22.064859 5040 generic.go:356] "Generic (PLEG): container finished" podID="810e3c9c-47dd-4e5b-a09f-72ba3c675e03" containerID="6a8f4895ec073157d4af1709c79d53cbecfa32dec1e1108ebeab52680fa57925" exitCode=0 Nov 01 07:25:22.065341 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:22.064926 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87c717b-7jndl" event={"ID":"810e3c9c-47dd-4e5b-a09f-72ba3c675e03","Type":"ContainerDied","Data":"6a8f4895ec073157d4af1709c79d53cbecfa32dec1e1108ebeab52680fa57925"} Nov 01 07:25:22.519298 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:22.519272 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 07:25:22.537602 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:22.537578 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 07:25:23.206970 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:23.206943 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87c717b-7jndl" Nov 01 07:25:23.301579 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:23.301545 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/810e3c9c-47dd-4e5b-a09f-72ba3c675e03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/810e3c9c-47dd-4e5b-a09f-72ba3c675e03-maas-api-openshift-ca\") pod \"810e3c9c-47dd-4e5b-a09f-72ba3c675e03\" (UID: \"810e3c9c-47dd-4e5b-a09f-72ba3c675e03\") " Nov 01 07:25:23.301769 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:23.301599 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/810e3c9c-47dd-4e5b-a09f-72ba3c675e03-kube-api-access-dnqg2\" (UniqueName: \"kubernetes.io/projected/810e3c9c-47dd-4e5b-a09f-72ba3c675e03-kube-api-access-dnqg2\") pod \"810e3c9c-47dd-4e5b-a09f-72ba3c675e03\" (UID: \"810e3c9c-47dd-4e5b-a09f-72ba3c675e03\") " Nov 01 07:25:23.301769 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:23.301648 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/810e3c9c-47dd-4e5b-a09f-72ba3c675e03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/810e3c9c-47dd-4e5b-a09f-72ba3c675e03-maas-api-xks-ca\") pod \"810e3c9c-47dd-4e5b-a09f-72ba3c675e03\" (UID: \"810e3c9c-47dd-4e5b-a09f-72ba3c675e03\") " Nov 01 07:25:23.301947 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:23.301922 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/810e3c9c-47dd-4e5b-a09f-72ba3c675e03-maas-api-openshift-ca" pod "810e3c9c-47dd-4e5b-a09f-72ba3c675e03" (UID: "810e3c9c-47dd-4e5b-a09f-72ba3c675e03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:25:23.303802 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:23.303776 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/810e3c9c-47dd-4e5b-a09f-72ba3c675e03-maas-api-xks-ca" pod "810e3c9c-47dd-4e5b-a09f-72ba3c675e03" (UID: "810e3c9c-47dd-4e5b-a09f-72ba3c675e03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:25:23.303919 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:23.303884 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/810e3c9c-47dd-4e5b-a09f-72ba3c675e03-kube-api-access-dnqg2" pod "810e3c9c-47dd-4e5b-a09f-72ba3c675e03" (UID: "810e3c9c-47dd-4e5b-a09f-72ba3c675e03"). InnerVolumeSpecName "kube-api-access-dnqg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:25:23.403383 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:23.403299 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/810e3c9c-47dd-4e5b-a09f-72ba3c675e03-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:25:23.403383 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:23.403333 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnqg2\" (UniqueName: \"kubernetes.io/projected/810e3c9c-47dd-4e5b-a09f-72ba3c675e03-kube-api-access-dnqg2\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:25:23.403383 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:23.403344 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/810e3c9c-47dd-4e5b-a09f-72ba3c675e03-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:25:24.076977 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:24.076936 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87c717b-7jndl" event={"ID":"810e3c9c-47dd-4e5b-a09f-72ba3c675e03","Type":"ContainerDied","Data":"0ed6aa4599a6219a5e783e3a6ec8e9ee81cb7bd132ad309e6d9e7adb9296848c"} Nov 01 07:25:24.077168 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:24.076993 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ed6aa4599a6219a5e783e3a6ec8e9ee81cb7bd132ad309e6d9e7adb9296848c" Nov 01 07:25:24.077168 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:24.076958 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87c717b-7jndl" Nov 01 07:25:39.977215 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:25:39.977171 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:25:55.387183 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:55.387146 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eba49bbc-qhkrn"] Nov 01 07:25:55.387679 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:55.387576 5040 cpu_manager.go:414] "containerMap: removing container" podUID="810e3c9c-47dd-4e5b-a09f-72ba3c675e03" containerName="revoke-keys" Nov 01 07:25:55.387679 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:55.387591 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="810e3c9c-47dd-4e5b-a09f-72ba3c675e03" containerName="revoke-keys" Nov 01 07:25:55.387679 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:55.387647 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="810e3c9c-47dd-4e5b-a09f-72ba3c675e03" containerName="revoke-keys" Nov 01 07:25:55.390863 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:55.390836 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba49bbc-qhkrn" Nov 01 07:25:55.445272 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:55.445237 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 07:25:55.445931 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:55.445896 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 07:25:55.449793 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:55.449768 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eba49bbc-qhkrn"] Nov 01 07:25:55.495428 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:55.495394 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 07:25:55.495620 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:55.495526 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cf2w\"" Nov 01 07:25:55.510601 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:55.510571 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/805c7036-07bf-4c15-bfed-52c9422a2f0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eba49bbc-qhkrn\" (UID: \"805c7036-07bf-4c15-bfed-52c9422a2f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba49bbc-qhkrn" Nov 01 07:25:55.510741 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:55.510613 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/805c7036-07bf-4c15-bfed-52c9422a2f0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eba49bbc-qhkrn\" (UID: \"805c7036-07bf-4c15-bfed-52c9422a2f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba49bbc-qhkrn" Nov 01 07:25:55.510792 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:55.510741 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwmsv\" (UniqueName: \"kubernetes.io/projected/805c7036-07bf-4c15-bfed-52c9422a2f0d-kube-api-access-xwmsv\") pod \"maas-api-revoke-sub-eba49bbc-qhkrn\" (UID: \"805c7036-07bf-4c15-bfed-52c9422a2f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba49bbc-qhkrn" Nov 01 07:25:55.612132 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:55.612091 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/805c7036-07bf-4c15-bfed-52c9422a2f0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eba49bbc-qhkrn\" (UID: \"805c7036-07bf-4c15-bfed-52c9422a2f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba49bbc-qhkrn" Nov 01 07:25:55.612334 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:55.612140 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/805c7036-07bf-4c15-bfed-52c9422a2f0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eba49bbc-qhkrn\" (UID: \"805c7036-07bf-4c15-bfed-52c9422a2f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba49bbc-qhkrn" Nov 01 07:25:55.612334 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:55.612183 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwmsv\" (UniqueName: \"kubernetes.io/projected/805c7036-07bf-4c15-bfed-52c9422a2f0d-kube-api-access-xwmsv\") pod \"maas-api-revoke-sub-eba49bbc-qhkrn\" (UID: \"805c7036-07bf-4c15-bfed-52c9422a2f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba49bbc-qhkrn" Nov 01 07:25:55.612841 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:55.612815 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/805c7036-07bf-4c15-bfed-52c9422a2f0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eba49bbc-qhkrn\" (UID: \"805c7036-07bf-4c15-bfed-52c9422a2f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba49bbc-qhkrn" Nov 01 07:25:55.614570 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:55.614547 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/805c7036-07bf-4c15-bfed-52c9422a2f0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eba49bbc-qhkrn\" (UID: \"805c7036-07bf-4c15-bfed-52c9422a2f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba49bbc-qhkrn" Nov 01 07:25:55.674287 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:55.674207 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwmsv\" (UniqueName: \"kubernetes.io/projected/805c7036-07bf-4c15-bfed-52c9422a2f0d-kube-api-access-xwmsv\") pod \"maas-api-revoke-sub-eba49bbc-qhkrn\" (UID: \"805c7036-07bf-4c15-bfed-52c9422a2f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba49bbc-qhkrn" Nov 01 07:25:55.703797 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:55.703761 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba49bbc-qhkrn" Nov 01 07:25:56.076020 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:56.075982 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eba49bbc-qhkrn"] Nov 01 07:25:56.077544 ip-10-0-143-195 kubenswrapper[5040]: W1101 07:25:56.077475 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod805c7036_07bf_4c15_bfed_52c9422a2f0d.slice/crio-eae0ea40e03e85bed980f73e4ee2fae1d70656f44e66c0a11a4a6d55211d46bc WatchSource:0}: Error finding container eae0ea40e03e85bed980f73e4ee2fae1d70656f44e66c0a11a4a6d55211d46bc: Status 404 returned error can't find the container with id eae0ea40e03e85bed980f73e4ee2fae1d70656f44e66c0a11a4a6d55211d46bc Nov 01 07:25:56.253385 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:56.253351 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba49bbc-qhkrn" event={"ID":"805c7036-07bf-4c15-bfed-52c9422a2f0d","Type":"ContainerStarted","Data":"eae0ea40e03e85bed980f73e4ee2fae1d70656f44e66c0a11a4a6d55211d46bc"} Nov 01 07:25:57.260197 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:57.260159 5040 generic.go:356] "Generic (PLEG): container finished" podID="805c7036-07bf-4c15-bfed-52c9422a2f0d" containerID="6c669d01f495d2b3c65986a4e5607bd6a1d31b19b8c63c23d4318ee7c3c3d0db" exitCode=0 Nov 01 07:25:57.260662 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:57.260207 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba49bbc-qhkrn" event={"ID":"805c7036-07bf-4c15-bfed-52c9422a2f0d","Type":"ContainerDied","Data":"6c669d01f495d2b3c65986a4e5607bd6a1d31b19b8c63c23d4318ee7c3c3d0db"} Nov 01 07:25:58.397720 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:58.397689 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba49bbc-qhkrn" Nov 01 07:25:58.541272 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:58.541186 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/805c7036-07bf-4c15-bfed-52c9422a2f0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/805c7036-07bf-4c15-bfed-52c9422a2f0d-maas-api-openshift-ca\") pod \"805c7036-07bf-4c15-bfed-52c9422a2f0d\" (UID: \"805c7036-07bf-4c15-bfed-52c9422a2f0d\") " Nov 01 07:25:58.541429 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:58.541328 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/805c7036-07bf-4c15-bfed-52c9422a2f0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/805c7036-07bf-4c15-bfed-52c9422a2f0d-maas-api-xks-ca\") pod \"805c7036-07bf-4c15-bfed-52c9422a2f0d\" (UID: \"805c7036-07bf-4c15-bfed-52c9422a2f0d\") " Nov 01 07:25:58.541429 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:58.541356 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/805c7036-07bf-4c15-bfed-52c9422a2f0d-kube-api-access-xwmsv\" (UniqueName: \"kubernetes.io/projected/805c7036-07bf-4c15-bfed-52c9422a2f0d-kube-api-access-xwmsv\") pod \"805c7036-07bf-4c15-bfed-52c9422a2f0d\" (UID: \"805c7036-07bf-4c15-bfed-52c9422a2f0d\") " Nov 01 07:25:58.541871 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:58.541651 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/805c7036-07bf-4c15-bfed-52c9422a2f0d-maas-api-openshift-ca" pod "805c7036-07bf-4c15-bfed-52c9422a2f0d" (UID: "805c7036-07bf-4c15-bfed-52c9422a2f0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:25:58.544055 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:58.544030 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/805c7036-07bf-4c15-bfed-52c9422a2f0d-maas-api-xks-ca" pod "805c7036-07bf-4c15-bfed-52c9422a2f0d" (UID: "805c7036-07bf-4c15-bfed-52c9422a2f0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:25:58.544139 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:58.544034 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/805c7036-07bf-4c15-bfed-52c9422a2f0d-kube-api-access-xwmsv" pod "805c7036-07bf-4c15-bfed-52c9422a2f0d" (UID: "805c7036-07bf-4c15-bfed-52c9422a2f0d"). InnerVolumeSpecName "kube-api-access-xwmsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:25:58.642367 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:58.642331 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/805c7036-07bf-4c15-bfed-52c9422a2f0d-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:25:58.642367 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:58.642365 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwmsv\" (UniqueName: \"kubernetes.io/projected/805c7036-07bf-4c15-bfed-52c9422a2f0d-kube-api-access-xwmsv\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:25:58.642583 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:58.642380 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/805c7036-07bf-4c15-bfed-52c9422a2f0d-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:25:59.271667 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:59.271632 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba49bbc-qhkrn" Nov 01 07:25:59.271667 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:59.271653 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba49bbc-qhkrn" event={"ID":"805c7036-07bf-4c15-bfed-52c9422a2f0d","Type":"ContainerDied","Data":"eae0ea40e03e85bed980f73e4ee2fae1d70656f44e66c0a11a4a6d55211d46bc"} Nov 01 07:25:59.271893 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:25:59.271703 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eae0ea40e03e85bed980f73e4ee2fae1d70656f44e66c0a11a4a6d55211d46bc" Nov 01 07:26:09.972340 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:26:09.972297 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:26:36.778754 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:36.778677 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a707025-8kzh2"] Nov 01 07:26:36.779242 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:36.779063 5040 cpu_manager.go:414] "containerMap: removing container" podUID="805c7036-07bf-4c15-bfed-52c9422a2f0d" containerName="revoke-keys" Nov 01 07:26:36.779242 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:36.779076 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="805c7036-07bf-4c15-bfed-52c9422a2f0d" containerName="revoke-keys" Nov 01 07:26:36.779242 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:36.779162 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="805c7036-07bf-4c15-bfed-52c9422a2f0d" containerName="revoke-keys" Nov 01 07:26:36.782431 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:36.782412 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a707025-8kzh2" Nov 01 07:26:36.835581 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:36.835550 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 07:26:36.879097 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:36.879062 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nvrn\" (UniqueName: \"kubernetes.io/projected/8a2229c7-0120-49fd-bb26-432a7ed1fe6a-kube-api-access-6nvrn\") pod \"maas-api-revoke-sub-1a707025-8kzh2\" (UID: \"8a2229c7-0120-49fd-bb26-432a7ed1fe6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a707025-8kzh2" Nov 01 07:26:36.879271 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:36.879112 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a2229c7-0120-49fd-bb26-432a7ed1fe6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a707025-8kzh2\" (UID: \"8a2229c7-0120-49fd-bb26-432a7ed1fe6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a707025-8kzh2" Nov 01 07:26:36.879271 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:36.879232 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a2229c7-0120-49fd-bb26-432a7ed1fe6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a707025-8kzh2\" (UID: \"8a2229c7-0120-49fd-bb26-432a7ed1fe6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a707025-8kzh2" Nov 01 07:26:36.886444 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:36.886404 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 07:26:36.886444 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:36.886404 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 07:26:36.886677 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:36.886459 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cf2w\"" Nov 01 07:26:36.890013 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:36.889990 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a707025-8kzh2"] Nov 01 07:26:36.980164 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:36.980126 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a2229c7-0120-49fd-bb26-432a7ed1fe6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a707025-8kzh2\" (UID: \"8a2229c7-0120-49fd-bb26-432a7ed1fe6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a707025-8kzh2" Nov 01 07:26:36.980399 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:36.980233 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nvrn\" (UniqueName: \"kubernetes.io/projected/8a2229c7-0120-49fd-bb26-432a7ed1fe6a-kube-api-access-6nvrn\") pod \"maas-api-revoke-sub-1a707025-8kzh2\" (UID: \"8a2229c7-0120-49fd-bb26-432a7ed1fe6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a707025-8kzh2" Nov 01 07:26:36.980399 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:36.980254 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a2229c7-0120-49fd-bb26-432a7ed1fe6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a707025-8kzh2\" (UID: \"8a2229c7-0120-49fd-bb26-432a7ed1fe6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a707025-8kzh2" Nov 01 07:26:36.980930 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:36.980908 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a2229c7-0120-49fd-bb26-432a7ed1fe6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a707025-8kzh2\" (UID: \"8a2229c7-0120-49fd-bb26-432a7ed1fe6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a707025-8kzh2" Nov 01 07:26:36.982648 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:36.982628 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a2229c7-0120-49fd-bb26-432a7ed1fe6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a707025-8kzh2\" (UID: \"8a2229c7-0120-49fd-bb26-432a7ed1fe6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a707025-8kzh2" Nov 01 07:26:37.041903 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:37.041843 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nvrn\" (UniqueName: \"kubernetes.io/projected/8a2229c7-0120-49fd-bb26-432a7ed1fe6a-kube-api-access-6nvrn\") pod \"maas-api-revoke-sub-1a707025-8kzh2\" (UID: \"8a2229c7-0120-49fd-bb26-432a7ed1fe6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a707025-8kzh2" Nov 01 07:26:37.095084 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:37.095055 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a707025-8kzh2" Nov 01 07:26:37.467588 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:37.467561 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a707025-8kzh2"] Nov 01 07:26:37.469385 ip-10-0-143-195 kubenswrapper[5040]: W1101 07:26:37.469357 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a2229c7_0120_49fd_bb26_432a7ed1fe6a.slice/crio-07679f0eecb4697ba9f1ff8448617517486ae7d33b460067d13d759a9f075ad1 WatchSource:0}: Error finding container 07679f0eecb4697ba9f1ff8448617517486ae7d33b460067d13d759a9f075ad1: Status 404 returned error can't find the container with id 07679f0eecb4697ba9f1ff8448617517486ae7d33b460067d13d759a9f075ad1 Nov 01 07:26:37.534948 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:37.534915 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a707025-8kzh2" event={"ID":"8a2229c7-0120-49fd-bb26-432a7ed1fe6a","Type":"ContainerStarted","Data":"07679f0eecb4697ba9f1ff8448617517486ae7d33b460067d13d759a9f075ad1"} Nov 01 07:26:38.542805 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:38.542763 5040 generic.go:356] "Generic (PLEG): container finished" podID="8a2229c7-0120-49fd-bb26-432a7ed1fe6a" containerID="5bb41fdb93f50e0c531fdfe3922d18b0528b6c2de889d15ae0a30bfad3fdbebe" exitCode=0 Nov 01 07:26:38.543294 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:38.542848 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a707025-8kzh2" event={"ID":"8a2229c7-0120-49fd-bb26-432a7ed1fe6a","Type":"ContainerDied","Data":"5bb41fdb93f50e0c531fdfe3922d18b0528b6c2de889d15ae0a30bfad3fdbebe"} Nov 01 07:26:39.698362 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:39.698332 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a707025-8kzh2" Nov 01 07:26:39.703561 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:39.703537 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a2229c7-0120-49fd-bb26-432a7ed1fe6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a2229c7-0120-49fd-bb26-432a7ed1fe6a-maas-api-openshift-ca\") pod \"8a2229c7-0120-49fd-bb26-432a7ed1fe6a\" (UID: \"8a2229c7-0120-49fd-bb26-432a7ed1fe6a\") " Nov 01 07:26:39.703687 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:39.703593 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a2229c7-0120-49fd-bb26-432a7ed1fe6a-kube-api-access-6nvrn\" (UniqueName: \"kubernetes.io/projected/8a2229c7-0120-49fd-bb26-432a7ed1fe6a-kube-api-access-6nvrn\") pod \"8a2229c7-0120-49fd-bb26-432a7ed1fe6a\" (UID: \"8a2229c7-0120-49fd-bb26-432a7ed1fe6a\") " Nov 01 07:26:39.703754 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:39.703689 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a2229c7-0120-49fd-bb26-432a7ed1fe6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a2229c7-0120-49fd-bb26-432a7ed1fe6a-maas-api-xks-ca\") pod \"8a2229c7-0120-49fd-bb26-432a7ed1fe6a\" (UID: \"8a2229c7-0120-49fd-bb26-432a7ed1fe6a\") " Nov 01 07:26:39.703855 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:39.703832 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a2229c7-0120-49fd-bb26-432a7ed1fe6a-maas-api-openshift-ca" pod "8a2229c7-0120-49fd-bb26-432a7ed1fe6a" (UID: "8a2229c7-0120-49fd-bb26-432a7ed1fe6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:26:39.703975 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:39.703959 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a2229c7-0120-49fd-bb26-432a7ed1fe6a-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:26:39.705663 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:39.705638 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a2229c7-0120-49fd-bb26-432a7ed1fe6a-kube-api-access-6nvrn" pod "8a2229c7-0120-49fd-bb26-432a7ed1fe6a" (UID: "8a2229c7-0120-49fd-bb26-432a7ed1fe6a"). InnerVolumeSpecName "kube-api-access-6nvrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:26:39.705747 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:39.705705 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a2229c7-0120-49fd-bb26-432a7ed1fe6a-maas-api-xks-ca" pod "8a2229c7-0120-49fd-bb26-432a7ed1fe6a" (UID: "8a2229c7-0120-49fd-bb26-432a7ed1fe6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:26:39.804531 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:39.804432 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a2229c7-0120-49fd-bb26-432a7ed1fe6a-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:26:39.804531 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:39.804464 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nvrn\" (UniqueName: \"kubernetes.io/projected/8a2229c7-0120-49fd-bb26-432a7ed1fe6a-kube-api-access-6nvrn\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:26:39.973687 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:26:39.973645 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:26:40.554368 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:40.554336 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a707025-8kzh2" Nov 01 07:26:40.554593 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:40.554365 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a707025-8kzh2" event={"ID":"8a2229c7-0120-49fd-bb26-432a7ed1fe6a","Type":"ContainerDied","Data":"07679f0eecb4697ba9f1ff8448617517486ae7d33b460067d13d759a9f075ad1"} Nov 01 07:26:40.554593 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:40.554413 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07679f0eecb4697ba9f1ff8448617517486ae7d33b460067d13d759a9f075ad1" Nov 01 07:26:57.515345 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:57.515312 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18f0701e-zc79j"] Nov 01 07:26:57.515916 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:57.515704 5040 cpu_manager.go:414] "containerMap: removing container" podUID="8a2229c7-0120-49fd-bb26-432a7ed1fe6a" containerName="revoke-keys" Nov 01 07:26:57.515916 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:57.515717 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a2229c7-0120-49fd-bb26-432a7ed1fe6a" containerName="revoke-keys" Nov 01 07:26:57.515916 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:57.515800 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a2229c7-0120-49fd-bb26-432a7ed1fe6a" containerName="revoke-keys" Nov 01 07:26:57.518923 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:57.518902 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f0701e-zc79j" Nov 01 07:26:57.556894 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:57.556859 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa021029-2df1-4b89-b75b-9f491dc8f186-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18f0701e-zc79j\" (UID: \"fa021029-2df1-4b89-b75b-9f491dc8f186\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f0701e-zc79j" Nov 01 07:26:57.557080 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:57.557048 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa021029-2df1-4b89-b75b-9f491dc8f186-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18f0701e-zc79j\" (UID: \"fa021029-2df1-4b89-b75b-9f491dc8f186\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f0701e-zc79j" Nov 01 07:26:57.557156 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:57.557140 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvgr2\" (UniqueName: \"kubernetes.io/projected/fa021029-2df1-4b89-b75b-9f491dc8f186-kube-api-access-nvgr2\") pod \"maas-api-revoke-sub-18f0701e-zc79j\" (UID: \"fa021029-2df1-4b89-b75b-9f491dc8f186\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f0701e-zc79j" Nov 01 07:26:57.572101 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:57.572072 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 07:26:57.622541 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:57.622498 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 07:26:57.622723 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:57.622562 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 07:26:57.622723 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:57.622562 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cf2w\"" Nov 01 07:26:57.625157 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:57.625136 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18f0701e-zc79j"] Nov 01 07:26:57.658376 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:57.658346 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa021029-2df1-4b89-b75b-9f491dc8f186-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18f0701e-zc79j\" (UID: \"fa021029-2df1-4b89-b75b-9f491dc8f186\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f0701e-zc79j" Nov 01 07:26:57.658568 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:57.658390 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvgr2\" (UniqueName: \"kubernetes.io/projected/fa021029-2df1-4b89-b75b-9f491dc8f186-kube-api-access-nvgr2\") pod \"maas-api-revoke-sub-18f0701e-zc79j\" (UID: \"fa021029-2df1-4b89-b75b-9f491dc8f186\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f0701e-zc79j" Nov 01 07:26:57.658568 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:57.658430 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa021029-2df1-4b89-b75b-9f491dc8f186-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18f0701e-zc79j\" (UID: \"fa021029-2df1-4b89-b75b-9f491dc8f186\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f0701e-zc79j" Nov 01 07:26:57.659129 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:57.659107 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa021029-2df1-4b89-b75b-9f491dc8f186-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18f0701e-zc79j\" (UID: \"fa021029-2df1-4b89-b75b-9f491dc8f186\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f0701e-zc79j" Nov 01 07:26:57.660695 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:57.660675 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa021029-2df1-4b89-b75b-9f491dc8f186-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18f0701e-zc79j\" (UID: \"fa021029-2df1-4b89-b75b-9f491dc8f186\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f0701e-zc79j" Nov 01 07:26:57.720163 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:57.720128 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvgr2\" (UniqueName: \"kubernetes.io/projected/fa021029-2df1-4b89-b75b-9f491dc8f186-kube-api-access-nvgr2\") pod \"maas-api-revoke-sub-18f0701e-zc79j\" (UID: \"fa021029-2df1-4b89-b75b-9f491dc8f186\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f0701e-zc79j" Nov 01 07:26:57.831957 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:57.831865 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f0701e-zc79j" Nov 01 07:26:58.210223 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:58.210195 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18f0701e-zc79j"] Nov 01 07:26:58.211144 ip-10-0-143-195 kubenswrapper[5040]: W1101 07:26:58.211118 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa021029_2df1_4b89_b75b_9f491dc8f186.slice/crio-bb84e5fb3a952b91226f41359a1b550fe8ffa92dce7f9790b5fec3bf23f9eb54 WatchSource:0}: Error finding container bb84e5fb3a952b91226f41359a1b550fe8ffa92dce7f9790b5fec3bf23f9eb54: Status 404 returned error can't find the container with id bb84e5fb3a952b91226f41359a1b550fe8ffa92dce7f9790b5fec3bf23f9eb54 Nov 01 07:26:58.651546 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:58.651506 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f0701e-zc79j" event={"ID":"fa021029-2df1-4b89-b75b-9f491dc8f186","Type":"ContainerStarted","Data":"2a3e327733d0974a1aa878fbb00d9b96fa29dbfb19d5a107799026c52c0dfd79"} Nov 01 07:26:58.651996 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:58.651564 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f0701e-zc79j" event={"ID":"fa021029-2df1-4b89-b75b-9f491dc8f186","Type":"ContainerStarted","Data":"bb84e5fb3a952b91226f41359a1b550fe8ffa92dce7f9790b5fec3bf23f9eb54"} Nov 01 07:26:58.767450 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:58.767307 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f0701e-zc79j" podStartSLOduration=1.767289553 podStartE2EDuration="1.767289553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:26:58.767270124 +0000 UTC m=+6097.422387290" watchObservedRunningTime="2026-11-01 07:26:58.767289553 +0000 UTC m=+6097.422406718" Nov 01 07:26:59.657853 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:59.657814 5040 generic.go:356] "Generic (PLEG): container finished" podID="fa021029-2df1-4b89-b75b-9f491dc8f186" containerID="2a3e327733d0974a1aa878fbb00d9b96fa29dbfb19d5a107799026c52c0dfd79" exitCode=0 Nov 01 07:26:59.658236 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:26:59.657901 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f0701e-zc79j" event={"ID":"fa021029-2df1-4b89-b75b-9f491dc8f186","Type":"ContainerDied","Data":"2a3e327733d0974a1aa878fbb00d9b96fa29dbfb19d5a107799026c52c0dfd79"} Nov 01 07:27:00.797855 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:00.797823 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f0701e-zc79j" Nov 01 07:27:00.891575 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:00.891537 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa021029-2df1-4b89-b75b-9f491dc8f186-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa021029-2df1-4b89-b75b-9f491dc8f186-maas-api-xks-ca\") pod \"fa021029-2df1-4b89-b75b-9f491dc8f186\" (UID: \"fa021029-2df1-4b89-b75b-9f491dc8f186\") " Nov 01 07:27:00.891761 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:00.891616 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa021029-2df1-4b89-b75b-9f491dc8f186-kube-api-access-nvgr2\" (UniqueName: \"kubernetes.io/projected/fa021029-2df1-4b89-b75b-9f491dc8f186-kube-api-access-nvgr2\") pod \"fa021029-2df1-4b89-b75b-9f491dc8f186\" (UID: \"fa021029-2df1-4b89-b75b-9f491dc8f186\") " Nov 01 07:27:00.891761 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:00.891688 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa021029-2df1-4b89-b75b-9f491dc8f186-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa021029-2df1-4b89-b75b-9f491dc8f186-maas-api-openshift-ca\") pod \"fa021029-2df1-4b89-b75b-9f491dc8f186\" (UID: \"fa021029-2df1-4b89-b75b-9f491dc8f186\") " Nov 01 07:27:00.892083 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:00.892057 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa021029-2df1-4b89-b75b-9f491dc8f186-maas-api-openshift-ca" pod "fa021029-2df1-4b89-b75b-9f491dc8f186" (UID: "fa021029-2df1-4b89-b75b-9f491dc8f186"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:27:00.893758 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:00.893733 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa021029-2df1-4b89-b75b-9f491dc8f186-maas-api-xks-ca" pod "fa021029-2df1-4b89-b75b-9f491dc8f186" (UID: "fa021029-2df1-4b89-b75b-9f491dc8f186"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:27:00.893883 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:00.893813 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa021029-2df1-4b89-b75b-9f491dc8f186-kube-api-access-nvgr2" pod "fa021029-2df1-4b89-b75b-9f491dc8f186" (UID: "fa021029-2df1-4b89-b75b-9f491dc8f186"). InnerVolumeSpecName "kube-api-access-nvgr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:27:00.993448 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:00.993357 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa021029-2df1-4b89-b75b-9f491dc8f186-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:27:00.993448 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:00.993393 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa021029-2df1-4b89-b75b-9f491dc8f186-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:27:00.993448 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:00.993403 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvgr2\" (UniqueName: \"kubernetes.io/projected/fa021029-2df1-4b89-b75b-9f491dc8f186-kube-api-access-nvgr2\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:27:01.669494 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:01.669453 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f0701e-zc79j" Nov 01 07:27:01.669668 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:01.669449 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f0701e-zc79j" event={"ID":"fa021029-2df1-4b89-b75b-9f491dc8f186","Type":"ContainerDied","Data":"bb84e5fb3a952b91226f41359a1b550fe8ffa92dce7f9790b5fec3bf23f9eb54"} Nov 01 07:27:01.669668 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:01.669587 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb84e5fb3a952b91226f41359a1b550fe8ffa92dce7f9790b5fec3bf23f9eb54" Nov 01 07:27:09.972018 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:27:09.971976 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:27:19.160681 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:19.160629 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-jl24d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 07:27:19.161372 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:19.161337 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-jl24d" podUID="4fea2872-e620-4756-9ede-22f65d64cf03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 07:27:34.862933 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:34.862881 5040 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tnrtq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 07:27:34.863628 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:34.863574 5040 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnrtq" podUID="09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 07:27:35.069291 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:35.069250 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c1e9675-tllgm"] Nov 01 07:27:35.145624 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:35.145531 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c1e9675-tllgm"] Nov 01 07:27:35.514461 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:35.514422 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c850a03-796c-46df-8085-896ed87eb85c" path="/var/lib/kubelet/pods/2c850a03-796c-46df-8085-896ed87eb85c/volumes" Nov 01 07:27:39.972640 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:27:39.972603 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:27:42.335881 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:42.335845 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db0ba793-cfcxb"] Nov 01 07:27:42.336377 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:42.336266 5040 cpu_manager.go:414] "containerMap: removing container" podUID="fa021029-2df1-4b89-b75b-9f491dc8f186" containerName="revoke-keys" Nov 01 07:27:42.336377 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:42.336280 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa021029-2df1-4b89-b75b-9f491dc8f186" containerName="revoke-keys" Nov 01 07:27:42.336455 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:42.336380 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa021029-2df1-4b89-b75b-9f491dc8f186" containerName="revoke-keys" Nov 01 07:27:42.339539 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:42.339523 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ba793-cfcxb" Nov 01 07:27:42.441977 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:42.441941 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpgbj\" (UniqueName: \"kubernetes.io/projected/4aa22774-3945-46a5-9df3-ffd8215ab609-kube-api-access-wpgbj\") pod \"maas-api-revoke-sub-db0ba793-cfcxb\" (UID: \"4aa22774-3945-46a5-9df3-ffd8215ab609\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ba793-cfcxb" Nov 01 07:27:42.442163 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:42.442003 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aa22774-3945-46a5-9df3-ffd8215ab609-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db0ba793-cfcxb\" (UID: \"4aa22774-3945-46a5-9df3-ffd8215ab609\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ba793-cfcxb" Nov 01 07:27:42.442163 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:42.442111 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aa22774-3945-46a5-9df3-ffd8215ab609-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db0ba793-cfcxb\" (UID: \"4aa22774-3945-46a5-9df3-ffd8215ab609\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ba793-cfcxb" Nov 01 07:27:42.443952 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:42.443927 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 07:27:42.444055 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:42.443954 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 07:27:42.444055 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:42.443977 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 07:27:42.444055 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:42.444020 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cf2w\"" Nov 01 07:27:42.446452 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:42.446431 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db0ba793-cfcxb"] Nov 01 07:27:42.543433 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:42.543403 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aa22774-3945-46a5-9df3-ffd8215ab609-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db0ba793-cfcxb\" (UID: \"4aa22774-3945-46a5-9df3-ffd8215ab609\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ba793-cfcxb" Nov 01 07:27:42.543628 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:42.543510 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpgbj\" (UniqueName: \"kubernetes.io/projected/4aa22774-3945-46a5-9df3-ffd8215ab609-kube-api-access-wpgbj\") pod \"maas-api-revoke-sub-db0ba793-cfcxb\" (UID: \"4aa22774-3945-46a5-9df3-ffd8215ab609\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ba793-cfcxb" Nov 01 07:27:42.543628 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:42.543601 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aa22774-3945-46a5-9df3-ffd8215ab609-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db0ba793-cfcxb\" (UID: \"4aa22774-3945-46a5-9df3-ffd8215ab609\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ba793-cfcxb" Nov 01 07:27:42.544126 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:42.544094 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aa22774-3945-46a5-9df3-ffd8215ab609-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db0ba793-cfcxb\" (UID: \"4aa22774-3945-46a5-9df3-ffd8215ab609\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ba793-cfcxb" Nov 01 07:27:42.545877 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:42.545855 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aa22774-3945-46a5-9df3-ffd8215ab609-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db0ba793-cfcxb\" (UID: \"4aa22774-3945-46a5-9df3-ffd8215ab609\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ba793-cfcxb" Nov 01 07:27:42.605785 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:42.605718 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpgbj\" (UniqueName: \"kubernetes.io/projected/4aa22774-3945-46a5-9df3-ffd8215ab609-kube-api-access-wpgbj\") pod \"maas-api-revoke-sub-db0ba793-cfcxb\" (UID: \"4aa22774-3945-46a5-9df3-ffd8215ab609\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ba793-cfcxb" Nov 01 07:27:42.652452 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:42.652422 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ba793-cfcxb" Nov 01 07:27:43.025200 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:43.025175 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db0ba793-cfcxb"] Nov 01 07:27:43.026732 ip-10-0-143-195 kubenswrapper[5040]: W1101 07:27:43.026707 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aa22774_3945_46a5_9df3_ffd8215ab609.slice/crio-9def23dba18f4a9b69d4dd8f86f66705fa6c793fce07ad46bfd78653d64ee4bb WatchSource:0}: Error finding container 9def23dba18f4a9b69d4dd8f86f66705fa6c793fce07ad46bfd78653d64ee4bb: Status 404 returned error can't find the container with id 9def23dba18f4a9b69d4dd8f86f66705fa6c793fce07ad46bfd78653d64ee4bb Nov 01 07:27:44.017420 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:44.017384 5040 generic.go:356] "Generic (PLEG): container finished" podID="4aa22774-3945-46a5-9df3-ffd8215ab609" containerID="90c11604a75031168214ab8b36930aeacd52d63160a5c912f3a36860776ded8d" exitCode=0 Nov 01 07:27:44.017844 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:44.017491 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ba793-cfcxb" event={"ID":"4aa22774-3945-46a5-9df3-ffd8215ab609","Type":"ContainerDied","Data":"90c11604a75031168214ab8b36930aeacd52d63160a5c912f3a36860776ded8d"} Nov 01 07:27:44.017844 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:44.017542 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ba793-cfcxb" event={"ID":"4aa22774-3945-46a5-9df3-ffd8215ab609","Type":"ContainerStarted","Data":"9def23dba18f4a9b69d4dd8f86f66705fa6c793fce07ad46bfd78653d64ee4bb"} Nov 01 07:27:45.170069 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:45.170044 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ba793-cfcxb" Nov 01 07:27:45.267555 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:45.267519 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aa22774-3945-46a5-9df3-ffd8215ab609-kube-api-access-wpgbj\" (UniqueName: \"kubernetes.io/projected/4aa22774-3945-46a5-9df3-ffd8215ab609-kube-api-access-wpgbj\") pod \"4aa22774-3945-46a5-9df3-ffd8215ab609\" (UID: \"4aa22774-3945-46a5-9df3-ffd8215ab609\") " Nov 01 07:27:45.267758 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:45.267597 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aa22774-3945-46a5-9df3-ffd8215ab609-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aa22774-3945-46a5-9df3-ffd8215ab609-maas-api-openshift-ca\") pod \"4aa22774-3945-46a5-9df3-ffd8215ab609\" (UID: \"4aa22774-3945-46a5-9df3-ffd8215ab609\") " Nov 01 07:27:45.267758 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:45.267691 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4aa22774-3945-46a5-9df3-ffd8215ab609-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aa22774-3945-46a5-9df3-ffd8215ab609-maas-api-xks-ca\") pod \"4aa22774-3945-46a5-9df3-ffd8215ab609\" (UID: \"4aa22774-3945-46a5-9df3-ffd8215ab609\") " Nov 01 07:27:45.267963 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:45.267932 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aa22774-3945-46a5-9df3-ffd8215ab609-maas-api-openshift-ca" pod "4aa22774-3945-46a5-9df3-ffd8215ab609" (UID: "4aa22774-3945-46a5-9df3-ffd8215ab609"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:27:45.269666 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:45.269636 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aa22774-3945-46a5-9df3-ffd8215ab609-maas-api-xks-ca" pod "4aa22774-3945-46a5-9df3-ffd8215ab609" (UID: "4aa22774-3945-46a5-9df3-ffd8215ab609"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:27:45.269772 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:45.269716 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aa22774-3945-46a5-9df3-ffd8215ab609-kube-api-access-wpgbj" pod "4aa22774-3945-46a5-9df3-ffd8215ab609" (UID: "4aa22774-3945-46a5-9df3-ffd8215ab609"). InnerVolumeSpecName "kube-api-access-wpgbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:27:45.368468 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:45.368425 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpgbj\" (UniqueName: \"kubernetes.io/projected/4aa22774-3945-46a5-9df3-ffd8215ab609-kube-api-access-wpgbj\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:27:45.368468 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:45.368463 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aa22774-3945-46a5-9df3-ffd8215ab609-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:27:45.368468 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:45.368474 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aa22774-3945-46a5-9df3-ffd8215ab609-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:27:46.030553 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:46.030516 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ba793-cfcxb" Nov 01 07:27:46.030742 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:46.030519 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ba793-cfcxb" event={"ID":"4aa22774-3945-46a5-9df3-ffd8215ab609","Type":"ContainerDied","Data":"9def23dba18f4a9b69d4dd8f86f66705fa6c793fce07ad46bfd78653d64ee4bb"} Nov 01 07:27:46.030742 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:46.030683 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9def23dba18f4a9b69d4dd8f86f66705fa6c793fce07ad46bfd78653d64ee4bb" Nov 01 07:27:53.132890 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:53.132855 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37c8d8b1-rtsqb"] Nov 01 07:27:53.133405 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:53.133286 5040 cpu_manager.go:414] "containerMap: removing container" podUID="4aa22774-3945-46a5-9df3-ffd8215ab609" containerName="revoke-keys" Nov 01 07:27:53.133405 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:53.133297 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aa22774-3945-46a5-9df3-ffd8215ab609" containerName="revoke-keys" Nov 01 07:27:53.133405 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:53.133361 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aa22774-3945-46a5-9df3-ffd8215ab609" containerName="revoke-keys" Nov 01 07:27:53.136475 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:53.136454 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c8d8b1-rtsqb" Nov 01 07:27:53.237614 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:53.237583 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c0706cc-fb73-47b4-b02a-67ff53f3bffb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37c8d8b1-rtsqb\" (UID: \"0c0706cc-fb73-47b4-b02a-67ff53f3bffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c8d8b1-rtsqb" Nov 01 07:27:53.237788 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:53.237694 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z8px\" (UniqueName: \"kubernetes.io/projected/0c0706cc-fb73-47b4-b02a-67ff53f3bffb-kube-api-access-6z8px\") pod \"maas-api-revoke-sub-37c8d8b1-rtsqb\" (UID: \"0c0706cc-fb73-47b4-b02a-67ff53f3bffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c8d8b1-rtsqb" Nov 01 07:27:53.237839 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:53.237795 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c0706cc-fb73-47b4-b02a-67ff53f3bffb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37c8d8b1-rtsqb\" (UID: \"0c0706cc-fb73-47b4-b02a-67ff53f3bffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c8d8b1-rtsqb" Nov 01 07:27:53.241442 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:53.241424 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 07:27:53.241648 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:53.241632 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cf2w\"" Nov 01 07:27:53.241718 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:53.241639 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 07:27:53.241718 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:53.241690 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 07:27:53.244135 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:53.244114 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37c8d8b1-rtsqb"] Nov 01 07:27:53.338884 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:53.338838 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c0706cc-fb73-47b4-b02a-67ff53f3bffb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37c8d8b1-rtsqb\" (UID: \"0c0706cc-fb73-47b4-b02a-67ff53f3bffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c8d8b1-rtsqb" Nov 01 07:27:53.339066 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:53.338948 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z8px\" (UniqueName: \"kubernetes.io/projected/0c0706cc-fb73-47b4-b02a-67ff53f3bffb-kube-api-access-6z8px\") pod \"maas-api-revoke-sub-37c8d8b1-rtsqb\" (UID: \"0c0706cc-fb73-47b4-b02a-67ff53f3bffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c8d8b1-rtsqb" Nov 01 07:27:53.339066 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:53.339010 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c0706cc-fb73-47b4-b02a-67ff53f3bffb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37c8d8b1-rtsqb\" (UID: \"0c0706cc-fb73-47b4-b02a-67ff53f3bffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c8d8b1-rtsqb" Nov 01 07:27:53.339716 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:53.339696 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c0706cc-fb73-47b4-b02a-67ff53f3bffb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37c8d8b1-rtsqb\" (UID: \"0c0706cc-fb73-47b4-b02a-67ff53f3bffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c8d8b1-rtsqb" Nov 01 07:27:53.341251 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:53.341230 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c0706cc-fb73-47b4-b02a-67ff53f3bffb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37c8d8b1-rtsqb\" (UID: \"0c0706cc-fb73-47b4-b02a-67ff53f3bffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c8d8b1-rtsqb" Nov 01 07:27:53.402848 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:53.402770 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z8px\" (UniqueName: \"kubernetes.io/projected/0c0706cc-fb73-47b4-b02a-67ff53f3bffb-kube-api-access-6z8px\") pod \"maas-api-revoke-sub-37c8d8b1-rtsqb\" (UID: \"0c0706cc-fb73-47b4-b02a-67ff53f3bffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c8d8b1-rtsqb" Nov 01 07:27:53.451547 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:53.451512 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c8d8b1-rtsqb" Nov 01 07:27:53.825063 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:53.825030 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37c8d8b1-rtsqb"] Nov 01 07:27:53.826269 ip-10-0-143-195 kubenswrapper[5040]: W1101 07:27:53.826245 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c0706cc_fb73_47b4_b02a_67ff53f3bffb.slice/crio-d68449272c8680e77ff89a5bd7c79ff2b774c0950198f5e696803f0f6c363c33 WatchSource:0}: Error finding container d68449272c8680e77ff89a5bd7c79ff2b774c0950198f5e696803f0f6c363c33: Status 404 returned error can't find the container with id d68449272c8680e77ff89a5bd7c79ff2b774c0950198f5e696803f0f6c363c33 Nov 01 07:27:54.075257 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:54.075185 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c8d8b1-rtsqb" event={"ID":"0c0706cc-fb73-47b4-b02a-67ff53f3bffb","Type":"ContainerStarted","Data":"d68449272c8680e77ff89a5bd7c79ff2b774c0950198f5e696803f0f6c363c33"} Nov 01 07:27:55.082221 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:55.082181 5040 generic.go:356] "Generic (PLEG): container finished" podID="0c0706cc-fb73-47b4-b02a-67ff53f3bffb" containerID="3b83f1882dc0506079158cad98d2b0eef6dcf45db4ceb0f83f5ea5c81f9002a8" exitCode=0 Nov 01 07:27:55.082651 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:55.082267 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c8d8b1-rtsqb" event={"ID":"0c0706cc-fb73-47b4-b02a-67ff53f3bffb","Type":"ContainerDied","Data":"3b83f1882dc0506079158cad98d2b0eef6dcf45db4ceb0f83f5ea5c81f9002a8"} Nov 01 07:27:56.262011 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:56.261973 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c8d8b1-rtsqb" Nov 01 07:27:56.368630 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:56.368528 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c0706cc-fb73-47b4-b02a-67ff53f3bffb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c0706cc-fb73-47b4-b02a-67ff53f3bffb-maas-api-xks-ca\") pod \"0c0706cc-fb73-47b4-b02a-67ff53f3bffb\" (UID: \"0c0706cc-fb73-47b4-b02a-67ff53f3bffb\") " Nov 01 07:27:56.368828 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:56.368636 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c0706cc-fb73-47b4-b02a-67ff53f3bffb-kube-api-access-6z8px\" (UniqueName: \"kubernetes.io/projected/0c0706cc-fb73-47b4-b02a-67ff53f3bffb-kube-api-access-6z8px\") pod \"0c0706cc-fb73-47b4-b02a-67ff53f3bffb\" (UID: \"0c0706cc-fb73-47b4-b02a-67ff53f3bffb\") " Nov 01 07:27:56.368828 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:56.368736 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c0706cc-fb73-47b4-b02a-67ff53f3bffb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c0706cc-fb73-47b4-b02a-67ff53f3bffb-maas-api-openshift-ca\") pod \"0c0706cc-fb73-47b4-b02a-67ff53f3bffb\" (UID: \"0c0706cc-fb73-47b4-b02a-67ff53f3bffb\") " Nov 01 07:27:56.369151 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:56.369120 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c0706cc-fb73-47b4-b02a-67ff53f3bffb-maas-api-openshift-ca" pod "0c0706cc-fb73-47b4-b02a-67ff53f3bffb" (UID: "0c0706cc-fb73-47b4-b02a-67ff53f3bffb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:27:56.370748 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:56.370722 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c0706cc-fb73-47b4-b02a-67ff53f3bffb-maas-api-xks-ca" pod "0c0706cc-fb73-47b4-b02a-67ff53f3bffb" (UID: "0c0706cc-fb73-47b4-b02a-67ff53f3bffb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:27:56.371249 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:56.371232 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c0706cc-fb73-47b4-b02a-67ff53f3bffb-kube-api-access-6z8px" pod "0c0706cc-fb73-47b4-b02a-67ff53f3bffb" (UID: "0c0706cc-fb73-47b4-b02a-67ff53f3bffb"). InnerVolumeSpecName "kube-api-access-6z8px". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:27:56.473173 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:56.473052 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c0706cc-fb73-47b4-b02a-67ff53f3bffb-maas-api-xks-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:27:56.473173 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:56.473106 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z8px\" (UniqueName: \"kubernetes.io/projected/0c0706cc-fb73-47b4-b02a-67ff53f3bffb-kube-api-access-6z8px\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:27:56.473173 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:56.473129 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c0706cc-fb73-47b4-b02a-67ff53f3bffb-maas-api-openshift-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:27:57.151383 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:57.151332 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c8d8b1-rtsqb" event={"ID":"0c0706cc-fb73-47b4-b02a-67ff53f3bffb","Type":"ContainerDied","Data":"d68449272c8680e77ff89a5bd7c79ff2b774c0950198f5e696803f0f6c363c33"} Nov 01 07:27:57.151715 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:57.151679 5040 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d68449272c8680e77ff89a5bd7c79ff2b774c0950198f5e696803f0f6c363c33" Nov 01 07:27:57.151964 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:27:57.151950 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c8d8b1-rtsqb" Nov 01 07:28:09.973572 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:28:09.973532 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:28:34.533394 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:28:34.533359 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-558f494858-fkgrf_ef1482fa-e45b-4784-939f-f6e9aa480c0e/authorino/0.log" Nov 01 07:28:39.973209 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:28:39.973168 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:28:57.272259 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:28:57.272224 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-788tv_72a66fbc-a5fe-42f7-8362-fe8e2a62025f/manager/1.log" Nov 01 07:28:58.092403 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:28:58.092368 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-c24v5_d73d9f29-4a48-46bf-8352-868e7d913a1b/manager/0.log" Nov 01 07:28:58.332527 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:28:58.332494 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-x2lbs_1edd6e44-240f-4f06-9294-f04fe4fcda65/proxy/0.log" Nov 01 07:29:08.996917 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:29:08.996884 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-788tv_72a66fbc-a5fe-42f7-8362-fe8e2a62025f/manager/0.log" Nov 01 07:29:09.972663 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:29:09.972627 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:29:10.297052 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:29:10.297021 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-18f0701e-zc79j_fa021029-2df1-4b89-b75b-9f491dc8f186/revoke-keys/0.log" Nov 01 07:29:10.524727 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:29:10.524689 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-1a707025-8kzh2_8a2229c7-0120-49fd-bb26-432a7ed1fe6a/revoke-keys/0.log" Nov 01 07:29:10.757770 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:29:10.757744 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-37c8d8b1-rtsqb_0c0706cc-fb73-47b4-b02a-67ff53f3bffb/revoke-keys/0.log" Nov 01 07:29:10.987965 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:29:10.987935 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c87c717b-7jndl_810e3c9c-47dd-4e5b-a09f-72ba3c675e03/revoke-keys/0.log" Nov 01 07:29:11.447737 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:29:11.447708 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-db0ba793-cfcxb_4aa22774-3945-46a5-9df3-ffd8215ab609/revoke-keys/0.log" Nov 01 07:29:21.671688 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:29:21.671659 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-eba49bbc-qhkrn_805c7036-07bf-4c15-bfed-52c9422a2f0d/revoke-keys/0.log" Nov 01 07:29:21.901214 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:29:21.901179 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f7782503-m9dz7_336701aa-8260-4998-841a-8b95f48c013d/revoke-keys/0.log" Nov 01 07:29:24.106516 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:29:24.106459 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-558f494858-fkgrf_ef1482fa-e45b-4784-939f-f6e9aa480c0e/authorino/0.log" Nov 01 07:29:24.478251 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:29:24.478161 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-p4xhq_4e64dd96-b5ba-4115-b87d-ffa91d637530/manager/0.log" Nov 01 07:29:24.699394 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:29:24.699362 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-6c4qz_da0c8971-758e-4a48-9f10-4c90ed9b4f1a/manager/0.log" Nov 01 07:29:24.730014 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:29:24.729935 5040 scope.go:122] "RemoveContainer" containerID="2c0488de18069872cd29418cee3cf003ddc06da1a4849575eb81bef24fb74deb" Nov 01 07:29:25.834759 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:29:25.834727 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-p8pjd_58847028-6023-4f04-bd0d-9d6a4327b694/manager/0.log" Nov 01 07:29:37.425217 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:29:37.425135 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84b94dcd67-jsghm_5e90e7f9-4509-484e-be03-f470072753c7/kube-auth-proxy/0.log" Nov 01 07:29:38.626640 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:29:38.626612 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-f895b_2b85392e-5dd8-4539-93ac-e659fb9cdf2c/istio-proxy/0.log" Nov 01 07:29:39.076739 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:29:39.076708 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-jl24d_4fea2872-e620-4756-9ede-22f65d64cf03/istio-proxy/0.log" Nov 01 07:29:39.772051 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:29:39.772021 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-tnrtq_09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7/istio-proxy/0.log" Nov 01 07:29:39.972128 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:29:39.972092 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:29:51.061410 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:29:51.061378 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6c5b5d5bd5-776bc_fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78/router/0.log" Nov 01 07:29:53.753295 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:29:53.753266 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-t2gjl_21f54836-d1ed-4f87-8f26-7c2d939b2590/main/0.log" Nov 01 07:30:00.176104 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:00.176068 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891970-4nk4g"] Nov 01 07:30:00.176665 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:00.176506 5040 cpu_manager.go:414] "containerMap: removing container" podUID="0c0706cc-fb73-47b4-b02a-67ff53f3bffb" containerName="revoke-keys" Nov 01 07:30:00.176665 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:00.176519 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c0706cc-fb73-47b4-b02a-67ff53f3bffb" containerName="revoke-keys" Nov 01 07:30:00.176665 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:00.176600 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c0706cc-fb73-47b4-b02a-67ff53f3bffb" containerName="revoke-keys" Nov 01 07:30:00.179915 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:00.179899 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-4nk4g" Nov 01 07:30:00.233380 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:00.233345 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 07:30:00.233594 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:00.233386 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 07:30:00.283714 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:00.283676 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cf2w\"" Nov 01 07:30:00.286501 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:00.286469 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891970-4nk4g"] Nov 01 07:30:00.311024 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:00.310995 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq5nl\" (UniqueName: \"kubernetes.io/projected/a81d80f9-a908-4fec-b5a8-14d3d718f7ec-kube-api-access-jq5nl\") pod \"maas-api-key-cleanup-29891970-4nk4g\" (UID: \"a81d80f9-a908-4fec-b5a8-14d3d718f7ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-4nk4g" Nov 01 07:30:00.311171 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:00.311127 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a81d80f9-a908-4fec-b5a8-14d3d718f7ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891970-4nk4g\" (UID: \"a81d80f9-a908-4fec-b5a8-14d3d718f7ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-4nk4g" Nov 01 07:30:00.411887 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:00.411850 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a81d80f9-a908-4fec-b5a8-14d3d718f7ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891970-4nk4g\" (UID: \"a81d80f9-a908-4fec-b5a8-14d3d718f7ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-4nk4g" Nov 01 07:30:00.412078 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:00.411909 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq5nl\" (UniqueName: \"kubernetes.io/projected/a81d80f9-a908-4fec-b5a8-14d3d718f7ec-kube-api-access-jq5nl\") pod \"maas-api-key-cleanup-29891970-4nk4g\" (UID: \"a81d80f9-a908-4fec-b5a8-14d3d718f7ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-4nk4g" Nov 01 07:30:00.412547 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:00.412527 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a81d80f9-a908-4fec-b5a8-14d3d718f7ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891970-4nk4g\" (UID: \"a81d80f9-a908-4fec-b5a8-14d3d718f7ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-4nk4g" Nov 01 07:30:00.474057 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:00.473982 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq5nl\" (UniqueName: \"kubernetes.io/projected/a81d80f9-a908-4fec-b5a8-14d3d718f7ec-kube-api-access-jq5nl\") pod \"maas-api-key-cleanup-29891970-4nk4g\" (UID: \"a81d80f9-a908-4fec-b5a8-14d3d718f7ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-4nk4g" Nov 01 07:30:00.492991 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:00.492964 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-4nk4g" Nov 01 07:30:00.867210 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:00.867177 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891970-4nk4g"] Nov 01 07:30:00.868259 ip-10-0-143-195 kubenswrapper[5040]: W1101 07:30:00.868232 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda81d80f9_a908_4fec_b5a8_14d3d718f7ec.slice/crio-9159f027f35512d2cc85abeed0722a29381a4a4ddb52971fc3243a3ef439ac74 WatchSource:0}: Error finding container 9159f027f35512d2cc85abeed0722a29381a4a4ddb52971fc3243a3ef439ac74: Status 404 returned error can't find the container with id 9159f027f35512d2cc85abeed0722a29381a4a4ddb52971fc3243a3ef439ac74 Nov 01 07:30:01.860227 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:01.860187 5040 generic.go:356] "Generic (PLEG): container finished" podID="a81d80f9-a908-4fec-b5a8-14d3d718f7ec" containerID="747852746eb160f4fd7dc854d067ab2666c3a378a026ce10febd920c2d034b50" exitCode=60 Nov 01 07:30:01.860671 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:01.860235 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-4nk4g" event={"ID":"a81d80f9-a908-4fec-b5a8-14d3d718f7ec","Type":"ContainerDied","Data":"747852746eb160f4fd7dc854d067ab2666c3a378a026ce10febd920c2d034b50"} Nov 01 07:30:01.860671 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:01.860302 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-4nk4g" event={"ID":"a81d80f9-a908-4fec-b5a8-14d3d718f7ec","Type":"ContainerStarted","Data":"9159f027f35512d2cc85abeed0722a29381a4a4ddb52971fc3243a3ef439ac74"} Nov 01 07:30:01.860671 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:01.860605 5040 scope.go:122] "RemoveContainer" containerID="747852746eb160f4fd7dc854d067ab2666c3a378a026ce10febd920c2d034b50" Nov 01 07:30:02.868112 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:02.868025 5040 generic.go:356] "Generic (PLEG): container finished" podID="a81d80f9-a908-4fec-b5a8-14d3d718f7ec" containerID="c0c6f5f751fb8cc6c129f680ab0d249ac619d936892b491b18daf33d96df0afc" exitCode=60 Nov 01 07:30:02.868571 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:02.868112 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-4nk4g" event={"ID":"a81d80f9-a908-4fec-b5a8-14d3d718f7ec","Type":"ContainerDied","Data":"c0c6f5f751fb8cc6c129f680ab0d249ac619d936892b491b18daf33d96df0afc"} Nov 01 07:30:02.868571 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:02.868170 5040 scope.go:122] "RemoveContainer" containerID="747852746eb160f4fd7dc854d067ab2666c3a378a026ce10febd920c2d034b50" Nov 01 07:30:02.868571 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:02.868436 5040 scope.go:122] "RemoveContainer" containerID="c0c6f5f751fb8cc6c129f680ab0d249ac619d936892b491b18daf33d96df0afc" Nov 01 07:30:02.868749 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:30:02.868723 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891970-4nk4g_odh-ai-gateway-infra(a81d80f9-a908-4fec-b5a8-14d3d718f7ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-4nk4g" podUID="a81d80f9-a908-4fec-b5a8-14d3d718f7ec" Nov 01 07:30:03.876264 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:03.876233 5040 scope.go:122] "RemoveContainer" containerID="c0c6f5f751fb8cc6c129f680ab0d249ac619d936892b491b18daf33d96df0afc" Nov 01 07:30:03.876692 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:30:03.876453 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891970-4nk4g_odh-ai-gateway-infra(a81d80f9-a908-4fec-b5a8-14d3d718f7ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-4nk4g" podUID="a81d80f9-a908-4fec-b5a8-14d3d718f7ec" Nov 01 07:30:09.973454 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:30:09.973412 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:30:13.056946 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:13.056907 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7782503-m9dz7"] Nov 01 07:30:13.107432 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:13.107394 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7782503-m9dz7"] Nov 01 07:30:13.513647 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:13.513608 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="336701aa-8260-4998-841a-8b95f48c013d" path="/var/lib/kubelet/pods/336701aa-8260-4998-841a-8b95f48c013d/volumes" Nov 01 07:30:17.518070 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:17.518035 5040 scope.go:122] "RemoveContainer" containerID="c0c6f5f751fb8cc6c129f680ab0d249ac619d936892b491b18daf33d96df0afc" Nov 01 07:30:17.963205 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:17.963168 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-4nk4g" event={"ID":"a81d80f9-a908-4fec-b5a8-14d3d718f7ec","Type":"ContainerStarted","Data":"e3c047673d66e574a80501d65a766d3f827898d05e6b73952c58e6861d5e29c8"} Nov 01 07:30:18.081356 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:18.081250 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-4nk4g" podStartSLOduration=18.081237928 podStartE2EDuration="18.081237928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:30:18.079085897 +0000 UTC m=+6296.734203064" watchObservedRunningTime="2026-11-01 07:30:18.081237928 +0000 UTC m=+6296.736355094" Nov 01 07:30:18.970752 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:18.970715 5040 generic.go:356] "Generic (PLEG): container finished" podID="a81d80f9-a908-4fec-b5a8-14d3d718f7ec" containerID="e3c047673d66e574a80501d65a766d3f827898d05e6b73952c58e6861d5e29c8" exitCode=60 Nov 01 07:30:18.971191 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:18.970787 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-4nk4g" event={"ID":"a81d80f9-a908-4fec-b5a8-14d3d718f7ec","Type":"ContainerDied","Data":"e3c047673d66e574a80501d65a766d3f827898d05e6b73952c58e6861d5e29c8"} Nov 01 07:30:18.971191 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:18.970848 5040 scope.go:122] "RemoveContainer" containerID="c0c6f5f751fb8cc6c129f680ab0d249ac619d936892b491b18daf33d96df0afc" Nov 01 07:30:18.971191 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:18.971183 5040 scope.go:122] "RemoveContainer" containerID="e3c047673d66e574a80501d65a766d3f827898d05e6b73952c58e6861d5e29c8" Nov 01 07:30:18.971447 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:30:18.971420 5040 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891970-4nk4g_odh-ai-gateway-infra(a81d80f9-a908-4fec-b5a8-14d3d718f7ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-4nk4g" podUID="a81d80f9-a908-4fec-b5a8-14d3d718f7ec" Nov 01 07:30:19.098597 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:19.098564 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891970-4nk4g"] Nov 01 07:30:20.116259 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:20.116235 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-4nk4g" Nov 01 07:30:20.199522 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:20.199466 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a81d80f9-a908-4fec-b5a8-14d3d718f7ec-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a81d80f9-a908-4fec-b5a8-14d3d718f7ec-maas-api-service-ca\") pod \"a81d80f9-a908-4fec-b5a8-14d3d718f7ec\" (UID: \"a81d80f9-a908-4fec-b5a8-14d3d718f7ec\") " Nov 01 07:30:20.199702 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:20.199577 5040 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a81d80f9-a908-4fec-b5a8-14d3d718f7ec-kube-api-access-jq5nl\" (UniqueName: \"kubernetes.io/projected/a81d80f9-a908-4fec-b5a8-14d3d718f7ec-kube-api-access-jq5nl\") pod \"a81d80f9-a908-4fec-b5a8-14d3d718f7ec\" (UID: \"a81d80f9-a908-4fec-b5a8-14d3d718f7ec\") " Nov 01 07:30:20.199916 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:20.199887 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a81d80f9-a908-4fec-b5a8-14d3d718f7ec-maas-api-service-ca" pod "a81d80f9-a908-4fec-b5a8-14d3d718f7ec" (UID: "a81d80f9-a908-4fec-b5a8-14d3d718f7ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:30:20.201692 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:20.201672 5040 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a81d80f9-a908-4fec-b5a8-14d3d718f7ec-kube-api-access-jq5nl" pod "a81d80f9-a908-4fec-b5a8-14d3d718f7ec" (UID: "a81d80f9-a908-4fec-b5a8-14d3d718f7ec"). InnerVolumeSpecName "kube-api-access-jq5nl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:30:20.301039 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:20.300998 5040 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq5nl\" (UniqueName: \"kubernetes.io/projected/a81d80f9-a908-4fec-b5a8-14d3d718f7ec-kube-api-access-jq5nl\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:30:20.301039 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:20.301029 5040 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a81d80f9-a908-4fec-b5a8-14d3d718f7ec-maas-api-service-ca\") on node \"ip-10-0-143-195.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:30:20.985153 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:20.985122 5040 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-4nk4g" Nov 01 07:30:20.985372 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:20.985167 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-4nk4g" event={"ID":"a81d80f9-a908-4fec-b5a8-14d3d718f7ec","Type":"ContainerDied","Data":"9159f027f35512d2cc85abeed0722a29381a4a4ddb52971fc3243a3ef439ac74"} Nov 01 07:30:20.985372 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:20.985250 5040 scope.go:122] "RemoveContainer" containerID="e3c047673d66e574a80501d65a766d3f827898d05e6b73952c58e6861d5e29c8" Nov 01 07:30:21.151582 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:21.151544 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891970-4nk4g"] Nov 01 07:30:21.157569 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:21.157542 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891970-4nk4g"] Nov 01 07:30:21.514875 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:21.514841 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a81d80f9-a908-4fec-b5a8-14d3d718f7ec" path="/var/lib/kubelet/pods/a81d80f9-a908-4fec-b5a8-14d3d718f7ec/volumes" Nov 01 07:30:22.557427 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:22.557308 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 07:30:22.583608 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:22.575974 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-njjn4_580bebeb-9495-4541-8d1d-3bf6fef1a061/console-operator/2.log" Nov 01 07:30:24.048306 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:24.048262 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c87c717b-7jndl"] Nov 01 07:30:24.098498 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:24.098446 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c87c717b-7jndl"] Nov 01 07:30:25.515969 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:25.515924 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="810e3c9c-47dd-4e5b-a09f-72ba3c675e03" path="/var/lib/kubelet/pods/810e3c9c-47dd-4e5b-a09f-72ba3c675e03/volumes" Nov 01 07:30:39.973378 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:30:39.973332 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:30:59.053863 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:59.053827 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eba49bbc-qhkrn"] Nov 01 07:30:59.104363 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:59.104323 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eba49bbc-qhkrn"] Nov 01 07:30:59.515360 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:30:59.515327 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="805c7036-07bf-4c15-bfed-52c9422a2f0d" path="/var/lib/kubelet/pods/805c7036-07bf-4c15-bfed-52c9422a2f0d/volumes" Nov 01 07:31:03.797044 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:03.797000 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s25q4/must-gather-w6n8n"] Nov 01 07:31:03.797455 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:03.797390 5040 cpu_manager.go:414] "containerMap: removing container" podUID="a81d80f9-a908-4fec-b5a8-14d3d718f7ec" containerName="cleanup" Nov 01 07:31:03.797455 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:03.797402 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a81d80f9-a908-4fec-b5a8-14d3d718f7ec" containerName="cleanup" Nov 01 07:31:03.797455 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:03.797426 5040 cpu_manager.go:414] "containerMap: removing container" podUID="a81d80f9-a908-4fec-b5a8-14d3d718f7ec" containerName="cleanup" Nov 01 07:31:03.797455 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:03.797431 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a81d80f9-a908-4fec-b5a8-14d3d718f7ec" containerName="cleanup" Nov 01 07:31:03.797649 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:03.797520 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="a81d80f9-a908-4fec-b5a8-14d3d718f7ec" containerName="cleanup" Nov 01 07:31:03.797649 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:03.797533 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="a81d80f9-a908-4fec-b5a8-14d3d718f7ec" containerName="cleanup" Nov 01 07:31:03.797649 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:03.797598 5040 cpu_manager.go:414] "containerMap: removing container" podUID="a81d80f9-a908-4fec-b5a8-14d3d718f7ec" containerName="cleanup" Nov 01 07:31:03.797649 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:03.797603 5040 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a81d80f9-a908-4fec-b5a8-14d3d718f7ec" containerName="cleanup" Nov 01 07:31:03.797780 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:03.797670 5040 memory_manager.go:365] "RemoveStaleState removing state" podUID="a81d80f9-a908-4fec-b5a8-14d3d718f7ec" containerName="cleanup" Nov 01 07:31:03.800936 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:03.800915 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s25q4/must-gather-w6n8n" Nov 01 07:31:03.857003 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:03.856968 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s25q4/must-gather-w6n8n"] Nov 01 07:31:03.865845 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:03.865807 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s25q4\"/\"openshift-service-ca.crt\"" Nov 01 07:31:03.895312 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:03.895270 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp9w4\" (UniqueName: \"kubernetes.io/projected/39325e3a-dec3-4e41-ad87-d85a5823e4b9-kube-api-access-zp9w4\") pod \"must-gather-w6n8n\" (UID: \"39325e3a-dec3-4e41-ad87-d85a5823e4b9\") " pod="openshift-must-gather-s25q4/must-gather-w6n8n" Nov 01 07:31:03.895542 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:03.895422 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/39325e3a-dec3-4e41-ad87-d85a5823e4b9-must-gather-output\") pod \"must-gather-w6n8n\" (UID: \"39325e3a-dec3-4e41-ad87-d85a5823e4b9\") " pod="openshift-must-gather-s25q4/must-gather-w6n8n" Nov 01 07:31:03.904880 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:03.904854 5040 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-s25q4\"/\"default-dockercfg-9wmvv\"" Nov 01 07:31:03.904880 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:03.904854 5040 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s25q4\"/\"kube-root-ca.crt\"" Nov 01 07:31:03.996758 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:03.996716 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/39325e3a-dec3-4e41-ad87-d85a5823e4b9-must-gather-output\") pod \"must-gather-w6n8n\" (UID: \"39325e3a-dec3-4e41-ad87-d85a5823e4b9\") " pod="openshift-must-gather-s25q4/must-gather-w6n8n" Nov 01 07:31:03.996955 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:03.996880 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp9w4\" (UniqueName: \"kubernetes.io/projected/39325e3a-dec3-4e41-ad87-d85a5823e4b9-kube-api-access-zp9w4\") pod \"must-gather-w6n8n\" (UID: \"39325e3a-dec3-4e41-ad87-d85a5823e4b9\") " pod="openshift-must-gather-s25q4/must-gather-w6n8n" Nov 01 07:31:03.997161 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:03.997138 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/39325e3a-dec3-4e41-ad87-d85a5823e4b9-must-gather-output\") pod \"must-gather-w6n8n\" (UID: \"39325e3a-dec3-4e41-ad87-d85a5823e4b9\") " pod="openshift-must-gather-s25q4/must-gather-w6n8n" Nov 01 07:31:04.058180 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:04.058100 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp9w4\" (UniqueName: \"kubernetes.io/projected/39325e3a-dec3-4e41-ad87-d85a5823e4b9-kube-api-access-zp9w4\") pod \"must-gather-w6n8n\" (UID: \"39325e3a-dec3-4e41-ad87-d85a5823e4b9\") " pod="openshift-must-gather-s25q4/must-gather-w6n8n" Nov 01 07:31:04.115163 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:04.115130 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s25q4/must-gather-w6n8n" Nov 01 07:31:04.491991 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:04.491965 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s25q4/must-gather-w6n8n"] Nov 01 07:31:04.494078 ip-10-0-143-195 kubenswrapper[5040]: W1101 07:31:04.494049 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39325e3a_dec3_4e41_ad87_d85a5823e4b9.slice/crio-4c545c756370f9dd60d09d8146656c27423ecbd4305f29b253d4e9336bbefc90 WatchSource:0}: Error finding container 4c545c756370f9dd60d09d8146656c27423ecbd4305f29b253d4e9336bbefc90: Status 404 returned error can't find the container with id 4c545c756370f9dd60d09d8146656c27423ecbd4305f29b253d4e9336bbefc90 Nov 01 07:31:04.496444 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:04.496425 5040 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 07:31:05.225349 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:05.225308 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s25q4/must-gather-w6n8n" event={"ID":"39325e3a-dec3-4e41-ad87-d85a5823e4b9","Type":"ContainerStarted","Data":"4c545c756370f9dd60d09d8146656c27423ecbd4305f29b253d4e9336bbefc90"} Nov 01 07:31:06.232987 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:06.232947 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s25q4/must-gather-w6n8n" event={"ID":"39325e3a-dec3-4e41-ad87-d85a5823e4b9","Type":"ContainerStarted","Data":"f23ff3a6773a5e4753fd758e38c9c7b8a91eca6e5aa94a01b67131e46b6907d5"} Nov 01 07:31:06.232987 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:06.232989 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s25q4/must-gather-w6n8n" event={"ID":"39325e3a-dec3-4e41-ad87-d85a5823e4b9","Type":"ContainerStarted","Data":"ca4398ffb2604a8927ec5d36e9b277831e9d12b2716aa4139155bf69d02e8e3c"} Nov 01 07:31:06.349457 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:06.349399 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-s25q4/must-gather-w6n8n" podStartSLOduration=2.121719355 podStartE2EDuration="3.349382983s" totalImagesPullingTime="1.227663628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:31:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:31:06.348456948 +0000 UTC m=+6345.003574116" watchObservedRunningTime="2026-11-01 07:31:06.349382983 +0000 UTC m=+6345.004500149" Nov 01 07:31:09.974316 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:31:09.974273 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:31:13.408349 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:13.408308 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-r5l82_7dc7486d-3188-4bb8-b907-b653caa18660/global-pull-secret-syncer/0.log" Nov 01 07:31:13.636500 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:13.634104 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-788tv_72a66fbc-a5fe-42f7-8362-fe8e2a62025f/manager/0.log" Nov 01 07:31:14.192692 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:14.190238 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-788tv_72a66fbc-a5fe-42f7-8362-fe8e2a62025f/manager/1.log" Nov 01 07:31:14.493835 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:14.493752 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-b2659_5e70277c-fae5-4e5a-b3d0-743d7485d2b3/connection-checker/0.log" Nov 01 07:31:14.937552 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:14.937520 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-b2kmh_1cc9d6f4-202c-45bb-8a4c-4b8e936874de/connection-checker/0.log" Nov 01 07:31:15.412399 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:15.412369 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-ls8v2_31894c07-ba1e-4865-9481-58555f919bcc/connection-checker/0.log" Nov 01 07:31:16.366504 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:16.365866 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-c24v5_d73d9f29-4a48-46bf-8352-868e7d913a1b/manager/0.log" Nov 01 07:31:16.485113 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:16.485076 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-x2lbs_1edd6e44-240f-4f06-9294-f04fe4fcda65/proxy/0.log" Nov 01 07:31:16.762522 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:16.761642 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-n7mks_197fb18c-6703-4deb-93b3-0186d7ceed67/konnectivity-agent/0.log" Nov 01 07:31:17.997925 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:17.997888 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-195.us-west-1.compute.internal_c656c4baa8463ac8d24886e04ca99821/haproxy/0.log" Nov 01 07:31:20.152436 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:20.152400 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-788tv_72a66fbc-a5fe-42f7-8362-fe8e2a62025f/manager/0.log" Nov 01 07:31:20.376959 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:20.376925 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-788tv_72a66fbc-a5fe-42f7-8362-fe8e2a62025f/manager/1.log" Nov 01 07:31:22.561810 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:22.561773 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-c24v5_d73d9f29-4a48-46bf-8352-868e7d913a1b/manager/0.log" Nov 01 07:31:22.723248 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:22.723206 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-x2lbs_1edd6e44-240f-4f06-9294-f04fe4fcda65/proxy/0.log" Nov 01 07:31:22.998765 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:22.998731 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-558f494858-fkgrf_ef1482fa-e45b-4784-939f-f6e9aa480c0e/authorino/0.log" Nov 01 07:31:23.539902 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:23.539866 5040 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s25q4/perf-node-gather-daemonset-s4v9f"] Nov 01 07:31:23.555644 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:23.555588 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-s4v9f" Nov 01 07:31:23.599036 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:23.598979 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s25q4/perf-node-gather-daemonset-s4v9f"] Nov 01 07:31:23.611220 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:23.610283 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/87fa8ebb-56e6-4f81-a650-c1894c054479-proc\") pod \"perf-node-gather-daemonset-s4v9f\" (UID: \"87fa8ebb-56e6-4f81-a650-c1894c054479\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-s4v9f" Nov 01 07:31:23.611567 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:23.611539 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/87fa8ebb-56e6-4f81-a650-c1894c054479-sys\") pod \"perf-node-gather-daemonset-s4v9f\" (UID: \"87fa8ebb-56e6-4f81-a650-c1894c054479\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-s4v9f" Nov 01 07:31:23.611895 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:23.611848 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/87fa8ebb-56e6-4f81-a650-c1894c054479-lib-modules\") pod \"perf-node-gather-daemonset-s4v9f\" (UID: \"87fa8ebb-56e6-4f81-a650-c1894c054479\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-s4v9f" Nov 01 07:31:23.612268 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:23.612229 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/87fa8ebb-56e6-4f81-a650-c1894c054479-podres\") pod \"perf-node-gather-daemonset-s4v9f\" (UID: \"87fa8ebb-56e6-4f81-a650-c1894c054479\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-s4v9f" Nov 01 07:31:23.612722 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:23.612505 5040 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvwc8\" (UniqueName: \"kubernetes.io/projected/87fa8ebb-56e6-4f81-a650-c1894c054479-kube-api-access-fvwc8\") pod \"perf-node-gather-daemonset-s4v9f\" (UID: \"87fa8ebb-56e6-4f81-a650-c1894c054479\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-s4v9f" Nov 01 07:31:23.713790 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:23.713747 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/87fa8ebb-56e6-4f81-a650-c1894c054479-proc\") pod \"perf-node-gather-daemonset-s4v9f\" (UID: \"87fa8ebb-56e6-4f81-a650-c1894c054479\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-s4v9f" Nov 01 07:31:23.713790 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:23.713791 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/87fa8ebb-56e6-4f81-a650-c1894c054479-sys\") pod \"perf-node-gather-daemonset-s4v9f\" (UID: \"87fa8ebb-56e6-4f81-a650-c1894c054479\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-s4v9f" Nov 01 07:31:23.714058 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:23.713820 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/87fa8ebb-56e6-4f81-a650-c1894c054479-lib-modules\") pod \"perf-node-gather-daemonset-s4v9f\" (UID: \"87fa8ebb-56e6-4f81-a650-c1894c054479\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-s4v9f" Nov 01 07:31:23.714058 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:23.713845 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/87fa8ebb-56e6-4f81-a650-c1894c054479-podres\") pod \"perf-node-gather-daemonset-s4v9f\" (UID: \"87fa8ebb-56e6-4f81-a650-c1894c054479\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-s4v9f" Nov 01 07:31:23.714058 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:23.713865 5040 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvwc8\" (UniqueName: \"kubernetes.io/projected/87fa8ebb-56e6-4f81-a650-c1894c054479-kube-api-access-fvwc8\") pod \"perf-node-gather-daemonset-s4v9f\" (UID: \"87fa8ebb-56e6-4f81-a650-c1894c054479\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-s4v9f" Nov 01 07:31:23.714058 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:23.713882 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/87fa8ebb-56e6-4f81-a650-c1894c054479-sys\") pod \"perf-node-gather-daemonset-s4v9f\" (UID: \"87fa8ebb-56e6-4f81-a650-c1894c054479\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-s4v9f" Nov 01 07:31:23.714058 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:23.713883 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/87fa8ebb-56e6-4f81-a650-c1894c054479-proc\") pod \"perf-node-gather-daemonset-s4v9f\" (UID: \"87fa8ebb-56e6-4f81-a650-c1894c054479\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-s4v9f" Nov 01 07:31:23.714058 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:23.713949 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/87fa8ebb-56e6-4f81-a650-c1894c054479-lib-modules\") pod \"perf-node-gather-daemonset-s4v9f\" (UID: \"87fa8ebb-56e6-4f81-a650-c1894c054479\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-s4v9f" Nov 01 07:31:23.714058 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:23.713960 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/87fa8ebb-56e6-4f81-a650-c1894c054479-podres\") pod \"perf-node-gather-daemonset-s4v9f\" (UID: \"87fa8ebb-56e6-4f81-a650-c1894c054479\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-s4v9f" Nov 01 07:31:23.784055 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:23.784021 5040 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvwc8\" (UniqueName: \"kubernetes.io/projected/87fa8ebb-56e6-4f81-a650-c1894c054479-kube-api-access-fvwc8\") pod \"perf-node-gather-daemonset-s4v9f\" (UID: \"87fa8ebb-56e6-4f81-a650-c1894c054479\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-s4v9f" Nov 01 07:31:23.872283 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:23.872205 5040 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-s4v9f" Nov 01 07:31:24.197843 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:24.197763 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-p4xhq_4e64dd96-b5ba-4115-b87d-ffa91d637530/manager/0.log" Nov 01 07:31:24.252604 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:24.252571 5040 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s25q4/perf-node-gather-daemonset-s4v9f"] Nov 01 07:31:24.254185 ip-10-0-143-195 kubenswrapper[5040]: W1101 07:31:24.254158 5040 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod87fa8ebb_56e6_4f81_a650_c1894c054479.slice/crio-c09c7128c1179617d74288e20948009eb19bdd3b76d4e6eeba56e25b3f4e94e1 WatchSource:0}: Error finding container c09c7128c1179617d74288e20948009eb19bdd3b76d4e6eeba56e25b3f4e94e1: Status 404 returned error can't find the container with id c09c7128c1179617d74288e20948009eb19bdd3b76d4e6eeba56e25b3f4e94e1 Nov 01 07:31:24.410951 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:24.410913 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-s4v9f" event={"ID":"87fa8ebb-56e6-4f81-a650-c1894c054479","Type":"ContainerStarted","Data":"178375c984c6b4ce4aa4ec808a1357b03c18eab595ef4c21954b6eee44a69fd4"} Nov 01 07:31:24.411131 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:24.410966 5040 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-s4v9f" event={"ID":"87fa8ebb-56e6-4f81-a650-c1894c054479","Type":"ContainerStarted","Data":"c09c7128c1179617d74288e20948009eb19bdd3b76d4e6eeba56e25b3f4e94e1"} Nov 01 07:31:24.411131 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:24.410991 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-s4v9f" Nov 01 07:31:24.534685 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:24.534629 5040 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-s4v9f" podStartSLOduration=1.534616832 podStartE2EDuration="1.534616832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 07:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:31:24.53394492 +0000 UTC m=+6363.189062087" watchObservedRunningTime="2026-11-01 07:31:24.534616832 +0000 UTC m=+6363.189733996" Nov 01 07:31:24.619090 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:24.619044 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-6c4qz_da0c8971-758e-4a48-9f10-4c90ed9b4f1a/manager/0.log" Nov 01 07:31:24.765707 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:24.765667 5040 scope.go:122] "RemoveContainer" containerID="6a8f4895ec073157d4af1709c79d53cbecfa32dec1e1108ebeab52680fa57925" Nov 01 07:31:24.777294 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:24.775551 5040 scope.go:122] "RemoveContainer" containerID="c60a3fb1b1d9e2c4c6cae4c5604a9013d89be04c66ede1b3a2583b75cae3fb90" Nov 01 07:31:26.317157 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:26.317118 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-788tv_72a66fbc-a5fe-42f7-8362-fe8e2a62025f/manager/0.log" Nov 01 07:31:26.423408 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:26.423372 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-p8pjd_58847028-6023-4f04-bd0d-9d6a4327b694/manager/0.log" Nov 01 07:31:26.547339 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:26.547302 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-788tv_72a66fbc-a5fe-42f7-8362-fe8e2a62025f/manager/1.log" Nov 01 07:31:28.064528 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:28.064462 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-c24v5_d73d9f29-4a48-46bf-8352-868e7d913a1b/manager/0.log" Nov 01 07:31:28.217132 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:28.217103 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-x2lbs_1edd6e44-240f-4f06-9294-f04fe4fcda65/proxy/0.log" Nov 01 07:31:30.429283 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:30.429255 5040 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-s4v9f" Nov 01 07:31:31.963360 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:31.963321 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-9j8j4_dc9422c2-b041-4f6b-8495-961a9571c1e4/cluster-monitoring-operator/0.log" Nov 01 07:31:32.136049 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:32.136019 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84b94dcd67-jsghm_5e90e7f9-4509-484e-be03-f470072753c7/kube-auth-proxy/0.log" Nov 01 07:31:33.205582 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:33.205548 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-f895b_2b85392e-5dd8-4539-93ac-e659fb9cdf2c/istio-proxy/0.log" Nov 01 07:31:33.628955 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:33.628926 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-jl24d_4fea2872-e620-4756-9ede-22f65d64cf03/istio-proxy/0.log" Nov 01 07:31:34.272459 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:34.272425 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-tnrtq_09bdb0f6-3f64-4e4b-8d03-f23a3cc0d5d7/istio-proxy/0.log" Nov 01 07:31:34.492815 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:34.492785 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2k242_4137336d-3d4e-40e1-8d59-f32700970597/node-exporter/0.log" Nov 01 07:31:34.909104 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:34.909078 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2k242_4137336d-3d4e-40e1-8d59-f32700970597/kube-rbac-proxy/0.log" Nov 01 07:31:35.361626 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:35.361602 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2k242_4137336d-3d4e-40e1-8d59-f32700970597/init-textfile/0.log" Nov 01 07:31:35.372975 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:35.372952 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6c5b5d5bd5-776bc_fd6fec7b-3c49-4c83-a1bb-513fcdbdfd78/router/0.log" Nov 01 07:31:39.983884 ip-10-0-143-195 kubenswrapper[5040]: E1101 07:31:39.983837 5040 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54tr8" containerName="extractor" Nov 01 07:31:40.061690 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:40.061652 5040 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a707025-8kzh2"] Nov 01 07:31:40.111762 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:40.111698 5040 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a707025-8kzh2"] Nov 01 07:31:40.437395 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:40.437359 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-558f494858-fkgrf_ef1482fa-e45b-4784-939f-f6e9aa480c0e/authorino/0.log" Nov 01 07:31:40.547588 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:40.547552 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-p4xhq_4e64dd96-b5ba-4115-b87d-ffa91d637530/manager/0.log" Nov 01 07:31:40.762209 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:40.762174 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-6c4qz_da0c8971-758e-4a48-9f10-4c90ed9b4f1a/manager/0.log" Nov 01 07:31:41.515056 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:41.515022 5040 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a2229c7-0120-49fd-bb26-432a7ed1fe6a" path="/var/lib/kubelet/pods/8a2229c7-0120-49fd-bb26-432a7ed1fe6a/volumes" Nov 01 07:31:41.672262 ip-10-0-143-195 kubenswrapper[5040]: I1101 07:31:41.672222 5040 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-p8pjd_58847028-6023-4f04-bd0d-9d6a4327b694/manager/0.log"